Faster image processing

ABSTRACT

Methods, medium, and machines associated with image processing performance are disclosed. Image data may be copied from input memory to main memory before performing CPU intensive operations, and efficiently copied from memory thereafter. CPU intensive operations may include decryption, decompression, image enhancement, or reformatting. Real time compression may be achieved by sub-sampling each frame. A separate plane compression aspect may provide for distinguishing between regions of an image, separating and/or masking the original image into multiple image planes, and compressing each separated image plane with a compression method that is optimal for its characteristics.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation in part of co-pending applicationSer. No. 11/820,300 and of application Ser. No. 09/758,573.

U.S. patent application Ser. No. 09/758,573, filed on Jan. 10, 2001, nowU.S. Pat. No. 7,671,864, issued Mar. 2, 2010, and entitled “FASTER IMAGEPROCESSING” is incorporated by reference.

U.S. patent application Ser. No. 11/820,300, filed on Jun. 18, 2007, andentitled “SEPARATE PLANE COMPRESSION USING A PLURALITY OF COMPRESSIONMETHODS INCLUDING ZLN AND ZLD METHODS” is incorporated by reference.

This application (through application Ser. No. 11/820,300), applicationSer. No. 11/820,300, and application Ser. No. 09/467,721, claim priorityunder 35 U.S.C. §119(e) of U.S. provisional application Ser. No.60/113,051, filed on Dec. 21, 1998, and entitled “METHODS OF ZERO LOSS(ZL) COMPRESSION AND ENCODING OF GRAYSCALE IMAGES”, which hereby isincorporated by reference.

U.S. patent application Ser. No. 09/467,721, filed on Dec. 20, 1999,entitled “VARIABLE GENERAL PURPOSE COMPRESSION FOR VIDEO IMAGES (ZLN)”,now U.S. Pat. No. 7,233,619, which hereby is incorporated by reference.

A continuation in part of application Ser. No. 09/467,721, filed Oct.27, 2005, entitled “HANDHELD VIDEO TRANSMISSION AND DISPLAY,”application Ser. No. 11/262,106, was published as U.S. publication2006/0114987, and is hereby incorporated by reference. Application Ser.No. 09/467,721 is now U.S. Pat. No. 7,233,619, issued Jun. 19, 2007.

A continuation in part of application Ser. No. 09/467,721, filed Dec.13, 2006, entitled “VARIABLE GENERAL PURPOSE COMPRESSION FOR VIDEOIMAGES (ZLN),” application Ser. No. 11/638,989, is hereby incorporatedby reference. Application Ser. No. 11/638,989 is now U.S. Pat. No.7,991,052 issued Aug. 2, 2011.

My U.S. patent application Ser. No. 09/470,566, filed on Dec. 22, 1999,and entitled GENERAL PURPOSE COMPRESSION FOR VIDEO IMAGES (RHN)“, knownas the “RHN” method, now U.S. Pat. No. 7,016,417, hereby is incorporatedby reference. The RHN application claims a priority date based on a U.S.provisional application Ser. No. 60/113,276 filed on Dec. 23, 1998,which also hereby is incorporated by reference.

This application further claims priority under 35 U.S.C. §119(e) of theco-pending U.S. provisional application Ser. No. 60/113,050 filed on1998 Dec. 21, and entitled “METHODS OF ADDING DOPPLER ENHANCEMENT TOGRAYSCALE COMPRESSION (ZLD).” The provisional application Ser. No.60/113,050 filed on Dec. 21, 1998 and entitled “METHODS OF ADDINGDOPPLER ENHANCEMENT TO GRAYSCALE COMPRESSION (ZLD)” is also herebyincorporated by reference.

My U.S. patent application Ser. No. 09/312,922, filed on May 17, 1999,entitled “SYSTEM FOR TRANSMITTING VIDEO IMAGES OVER A COMPUTER NETWORKTO A REMOTE RECEIVER, ” now U.S. Pat. No. 7,257,158, describes anembodiment of the invention of the RHN method, as well as a system forpracticing the compression method, and also hereby is incorporated byreference.

U.S. patent application Ser. No. 09/436,432, filed on Nov. 8, 1999, andentitled “SYSTEM FOR TRANSMITTING VIDEO IMAGES OVER A COMPUTER NETWORKTO A REMOTE RECEIVER,” now U.S. Pat. No. 7,191,462, is wholly owned bythe inventor of the present invention.

ZLN is a three-letter identifier used to refer to the family ofcompression methods disclosed in the ZLN application. ZLD is athree-letter identifier used to refer to the family of compressionsmethods disclosed herein. ZL originally stood for ZeroLoss, a trademarkof Kendyl Román referring to the clinically lossless nature of themethods. The N in ZLN refers to the variable nature of the method when Ncan be on of a plurality of values. The D in ZLD refers to the addedcapabilities for handling Doppler enhanced images in conjunction with acompression method such as one of the ZLN family of methods.

The ZLN and ZLD family of compression methods can be practiced on anynumber apparatus or medium known in the art, including those disclosedherein or in U.S. provisional application Ser. No. 60/085,818,international application serial number PCT/US99/10894, internationalpublication number WO 99/59472, U.S. application Ser No. 09/312,922 orU.S. patent application Ser. No. 09/436,432, U.S. Pat. No. 7,191,462.

U.S. patent application Ser. No. 09/433,978, filed on Nov. 4, 1999, andentitled “GRAPHICAL USER INTERFACE INCLUDING ZOOM CONTROL REPRESENTINGIMAGE AND MAGNIFICATION OF DISPLAYED IMAGE”, now U.S. Pat. No.6,803,931, is wholly owned by the inventor of the present invention.

BACKGROUND

1. Field of the Invention

This invention relates to image processing.

2. Related Technology

ANSI Standard C “memcpy” Function

A given computer hardware architecture will have an optimal means ofcopying a block of data from one location in a memory to anotherlocation. Complex Instruction Set Computing (CISC) architecturesimplement instructions that over a number of CPU cycles move a block ofdata. Reduced Instruction Set Computing (RISC) architectures optimizethe instruction set to process each instruction in one or two CPU cyclesbut also included instructions that can be used to implement a shortroutine that will accomplish the block move in an optimal manner. Anefficient routine for copying a block of data can be implemented foreach specific computer architecture.

Some computer architectures include Direct Memory Access (DMA) circuitrythat transfers data between memory and input/output (I/O) deviceswithout continual central processing unit (CPU) intervention.

The ANSI standard for the C Programming Language defines a “memcpy”library function as an interface to an efficient routine for copying ablock of bytes to another location.

Graphical Images

A television screen has a 4:3 aspect ratio. In the United States,television signals contain 525 scan lines of which 480 lines are visibleon most televisions. When an analog video signal is digitized, each ofthe 480 lines are sampled 640 times, and each sample is represented by anumber. Each sample point is called a picture element, or pixel. A twodimensional array is created that is 640 pixels wide and 480 pixelshigh. This 640×480 pixel array is a still graphical image that isconsidered to be full frame. The human eye can optimally perceiveapproximately 16.7 thousand colors. A pixel value comprised of 24 bitscan represent each perceivable color. A graphical image made up of24-bit pixels is considered to be full color. A standard Super VGA(SVGA) computer display has a screen resolution of 640 by 480 pixel.Twenty-four bits is three bytes. It is common to use a fourth byte foreach pixel to specify a mask value or alpha channel. A typical imagebeing processed may contain over 1.2 million bytes of data.

When digitizing a video signal, or when manipulating the graphics to beoutput as a video signal or to be displayed on a computer display it maybe necessary to copy the image data to another area of memory (a buffer)for some type of image processing. However, the copied buffer takes upsignificant memory resources. Also the time it takes to copy the imagecan be significant especially when the image processing must be done inreal time. Those skilled in the art realize that to improve processingperformance the number of memory buffers containing a copy of the samedata should be reduced to the minimum set possible.

Display Video RAM

The memory of a computer system may be physically implemented indifferent areas or on different boards. The main memory is used forstorage of program instructions and data. A special memory area called“video RAM” may be dedicated to storing the image that is to bedisplayed on the computer display. The video RAM has special hardwarethat allows it to be accessed to update the display over 60 times asecond.

Capture Video RAM

A video digitizer or video capture card may also contain a specialmemory area similar to display video RAM for capturing the digitalsamples from the video signal. This RAM may also have special hardwarethat allows it to be updated 60 times a second.

Cache Memory

Many computer architectures implement one or more levels of memorycaching whereby blocks of memory data are stored in a cache memory thatmay be accessed more rapidly by the CPU. Typically input and output(I/O) memories such as video RAM, capture RAM, or hard disk buffers arenot cached.

3. Description of Prior Art

In the last few years, there have been tremendous advances in the speedof computer processors and in the availability of bandwidth of worldwidecomputer networks such as the Internet. These advances have led to apoint where businesses and households now commonly have both thecomputing power and network connectivity necessary to havepoint-to-point digital communications of audio, rich graphical images,and video. However the transmission of video signals with the fullresolution and quality of television is still out of reach. In order toachieve an acceptable level of video quality, the video signal must becompressed significantly without losing either spatial or temporalquality.

A number of different approaches have been taken but each has resultedin less than acceptable results. These approaches and theirdisadvantages are disclosed by Mark Nelson in a book entitled The DataCompression Book. Second Edition, published by M&T Book in 1996. MarkMorrision also discusses the state of the art in a book entitled TheMagic of Image Processing, published by Sams Publishing in 1993.

Video Signals

Standard video signals are analog in nature. In the United States,television signals contain 525 scan lines of which 480 lines are visibleon most televisions. The video signal represents a continuous stream ofstill images, also known as frames, which are fully scanned, transmittedand displayed at a rate of 30 frames per second. This frame rate isconsidered full motion.

A television screen has a 4:3 aspect ratio.

When an analog video signal is digitized, each of the 480 lines issampled 640 times, and each sample is represented by a number. Eachsample point is called a picture element, or pixel. A two dimensionalarray is created that is 640 pixels wide and 480 pixels high. This640×480 pixel array is a still graphical image that is considered to befull frame. The human eye can perceive 16.7 thousand colors. A pixelvalue comprised of 24 bits can represent each perceivable color. Agraphical image made up of 24-bit pixels is considered to be full color.A single, second-long, full frame, full color video requires over 220millions bits of data.

The transmission of 640×480 pixels×24 bits per pixel times 30 framesrequires the transmission of 221,184,000 million bits per second. A T1Internet connection can transfer up to 1.54 million bits per second. Ahigh-speed (56 Kb) modem can transfer data at a maximum rate of 56thousand bits per second. The transfer of full motion, full frame, fullcolor digital video over a T1 Internet connection, or 56 Kb modem, willrequire an effective data compression of over 144:1, or 3949:1,respectively.

A video signal typically will contain some signal noise. In the casewhere the image is generated based on sampled data, such as anultrasound machine, there is often noise and artificial spikes in thesignal. A video signal recorded on magnetic tape may have fluctuationsdue the irregularities in the recording media. Florescent or improperlighting may cause a solid background to flicker or appear grainy. Suchnoise exists in the real world but may reduce the quality of theperceived image and lower the compression ratio that could be achievedby conventional methods.

Basic Run-Length Encoding

An early technique for data compression is run-length encoding where arepeated series of items are replaced with one sample item and a countfor the number of times the sample repeats. Prior art shows run-lengthencoding of both individual bits and bytes. These simple approaches bythemselves have failed to achieve the necessary compression ratios.

Variable Length Encoding

In the late 1940s, Claude Shannon at Bell Labs and R. M. Fano at MITpioneered the field of data compression. Their work resulted in atechnique of using variable length codes where codes with lowprobabilities have more bits, and codes with higher probabilities havefewer bits. This approach requires multiple passes through the data todetermine code probability and then to encode the data. This approachalso has failed to achieve the necessary compression ratios.

D. A. Huffman disclosed a more efficient approach of variable lengthencoding known as Huffman coding in a paper entitled “A Method forConstruction of Minimum Redundancy Codes,” published in 1952. Thisapproach also has failed to achieve the necessary compression ratios.

Arithmetic, Finite Context, and Adaptive Coding

In the 1980s, arithmetic, finite coding, and adaptive coding haveprovided a slight improvement over the earlier methods. These approachesrequire extensive computer processing and have failed to achieve thenecessary compression ratios.

Dictionary-Based Compression

Dictionary-based compression uses a completely different method tocompress data. Variable length strings of symbols are encoded as singletokens. The tokens form an index to a dictionary. In 1977, AbrahamLempel and Jacob Ziv published a paper entitled, “A Universal Algorithmfor Sequential Data Compression” in IEEE Transactions on InformationTheory, which disclosed a compression technique commonly known as LZ77.The same authors published a 1978 sequel entitled, “Compression ofIndividual Sequences via Variable-Rate Coding,” which disclosed acompression technique commonly known as LZ78 (see U.S. Pat. No.4,464,650). Terry Welch published an article entitled, “A Technique forHigh-Performance Data Compression,” in the June 1984 issue of IEEEComputer, which disclosed an algorithm commonly known as LZW, which isthe basis for the GIF algorithm (see U.S. Pat. Nos. 4,558,302,4,814,746, and 4,876,541). In 1989, Stack Electronics implemented a LZ77based method called QIC-122 (see U.S. Pat. Nos. 5,532,694, 5,506,580,and 5,463,390).

These lossless (method where no data is lost) compression methods canachieve up to 10:1 compression ratios on graphic images typical of avideo image. While these dictionary-based algorithms are popular, theseapproaches require extensive computer processing and have failed toachieve the necessary compression ratios.

JPEG and MPEG

Graphical images have an advantage over conventional computer datafiles: they can be slightly modified during thecompression/decompression cycle without affecting the perceived qualityon the part of the viewer. By allowing some loss of data, compressionratios of 25:1 have been achieved without major degradation of theperceived image. The Joint Photographic Experts Group (JPEG) hasdeveloped a standard for graphical image compression. The JPEG lossy(method where some data is lost) compression algorithm first divides thecolor image into three color planes and divides each plane into 8 by 8blocks, and then the algorithm operates in three successive stages:

-   -   (a) A mathematical transformation known as Discrete Cosine        Transform (DCT) takes a set of points from the spatial domain        and transforms them into an identical representation in the        frequency domain.    -   (b) A lossy quantization is performed using a quantization        matrix to reduce the precision of the coefficients.    -   (c) The zero values are encoded in a zig-zag sequence (see        Nelson, pp. 341-342).

JPEG can be scaled to perform higher compression ratio by allowing moreloss in the quantization stage of the compression. However this lossresults in certain blocks of the image being compressed such that areasof the image have a blocky appearance and the edges of the 8 by 8 blocksbecome apparent because they no longer match the colors of theiradjacent blocks. Another disadvantage of JPEG is smearing. The trueedges in an image get blurred due to the lossy compression method.

The Moving Pictures Expert Group (MPEG) uses a combination of JPEG basedtechniques combined with forward and reverse temporal differencing. MPEGcompares adjacent frames and, for those blocks that are identical tothose in a previous or subsequent frame, only a description of theprevious or subsequent identical block is encoded. MPEG suffers from thesame blocking and smearing problems as JPEG.

These approaches require extensive computer processing and have failedto achieve the necessary compression ratios without unacceptable loss ofimage quality and artificially induced distortion.

QuickTime: CinePak, Sorensen, H.263

Apple Computer, Inc. released a component architecture for digital videocompression and decompression, named QuickTime. Any number of methodscan be encoded into a QuickTime compressor/decompressor (codec). Somepopular codec are CinePak, Sorensen, and H.263. CinePak and Sorensenboth require extensive computer processing to prepare a digital videosequence for playback in real time; neither can be used for livecompression. H.263 compresses in real time but does so by sacrificingimage quality resulting in severe blocking and smearing.

Fractal and Wavelet Compression

Extremely high compression ratios are achievable with fractal andwavelet compression algorithms. These approaches require extensivecomputer processing and generally cannot be completed in real time.

Sub-Sampling

Sub-sampling is the selection of a subset of data from a larger set ofdata. For example, when every other pixel of every other row of a videoimage is selected, the resulting image has half the width and half theheight. This is image sub-sampling. Other types of sub-sampling includeframe sub-sampling, area sub-sampling, and bit-wise sub-sampling.

Image Stretching

If an image is to be enlarged but maintain the same number of pixels perinch, data must be filled in for the new pixels that are added. Variousmethods of stretching an image and filling in the new pixels to maintainimage consistency are known in the art. Some methods known in the artare dithering (using adjacent colors that appear to be blended color),and error diffusion, “nearest neighbor”, bilinear and bicubic.

Doppler Enhancement

Doppler techniques are used to determine the velocities of one or moresmall objects. Some common uses of Doppler techniques include withoutlimitation:

-   -   1. Radar used to detect rain    -   2. Radar used to determine speed of vehicles or aircraft    -   3. Ultrasound blood flow analysis

Doppler velocity scales are often incorporated with grayscale images.

In the case of ultrasound blood flow analysis, average velocities towardthe sensing probe are encoded as a shade of red and velocities away fromthe sensing probe are encoded as a shade of blue. Although the imageappears to be in color, there are really three monochromic values: agrayscale, a red scale, and a blue scale. The base image plane(grayscale ultrasound) is generated more often (typically 15-30 framesper second) than the overlay plane showing the Doppler red and bluescales (typically 3-10 frames per second).

In the case of rain, the base map of the earth is generated only onceand the Doppler colors that indicate the intensity of the precipitationare laid over the base map.

Moving Pictures

A video or movie is comprised of a series of still images that, whendisplayed in sequence, appear to the human eye as a live motion image.Each still image is called a frame. Television in the USA displaysframes at the rate of 30 frames per second. Theater motion pictures aredisplayed at 24 frames per second. Cartoon animation is typicallydisplayed at 8-12 frames per second.

Compression Methods

The ZLN and ZLD methods are effective ways to compress video images.Other compression algorithms are known in the prior art, including RLE,GIF (LZW), MPEG, Cinepak, Motion-JPEG, Sorensen, Fractal, and manyothers.

Each of these methods treats a frame of video as a basic unit ofcompression applying the compression method uniformly to the entireimage.

Color Plane Separation

It is well known in the art that an image can be uniformly separatedinto color planes based on the red, green, and blue components valuesfor each pixel, based on hue, saturation, and brightness componentvalues for each pixel, or based on ink colors, such as cyan, yellow,magenta, and black. However these color plane separations are not doneto reduce data size or to aid compression. They are used to facilitatethe display (such as on a RGB or YUV computer monitor) or the printingof the image (for example, four-color printing).

Frame Differencing

MPEG and some other compression methods compare adjacent frames in astream of frames. Under certain circumstances these methods send only asubset of a frame (namely a rectangular portion that contains a changewhen compared to the adjacent frame) which is then overlaid on theunchanged data for the adjacent frame.

SUMMARY OF THE INVENTION

In accordance with the present invention, methods are provided ofincreasing performance of image processing by copying image data betweenI/O memory and main memory where CPU intensive processing of the imagedata is more efficiently performed.

A further aspect of the present invention is a method of compression ofa video stream comprises steps of sub-sampling a video frame, andrun-length encoding the sub-sampled pixel values, whereby the method canbe executed in real time and the compressed representation of pixelssaves substantial space on a storage medium and requires substantiallyless time and bandwidth to be transported over a communications link.The present invention includes a corresponding method for decompressingthe encoded data.

The ZLD format relates specifically to the compression and decompressionof video images that contain a grayscale image overlaid with Dopplerenhancement.

The separate plane compression aspect of the invention relatesspecifically to the compression and decompression of video images thatcontain portions that can be separated from other portions to optimizecompression, storage, or transmission.

Doppler velocity scales are incorporated into grayscale compressionmethods using two bits.

In accordance with an aspect of the present invention, a method ofadding Doppler enhancement to compression code typically formatted forgrayscale only, by using two bits of the data field to represent thescale of the remaining bits where said bits indicate one of the set ofscales comprising:

1. grayscale,

2. red scale, and

3. blue scale.

The present invention teaches that, often in a video stream, a smallregion of the image is of greater interest to the viewer, or recipient,of the video stream. The region of interest is required at a highquality and high frame rate. The portion of the image that is not ofprimary interest is still important for reference and orientation andstill needs to be displayed but can be displayed at lower quality and atlower frame rate. Said portion is necessary so that the viewer can knowwhen to change the focus to a new region of interest.

The present invention also teaches that, in some cases, separate regionsare of similar or equal interest, but, because of distinctcharacteristics, the image can be separated into multiple regions thatcan be separated into planes. Upon separation the planes are compressedwith a method optimal to the distinguishing characteristics of eachplane, and transferred at a rate optimal to each compressed plane.

In accordance with an aspect of the present invention, a method ofdistinguishing between regions of an image, separating and masking theoriginal image into multiple image planes, and compressing eachseparated image plane with a compression method that is optimal for itscharacteristics. From a video stream, separate image streams can bestored or transmitted at different rates. Alternatively, framedifferencing can be applied to the separated streams.

One method of distinguishing the region of interest is use of an inputdevice to allow the viewer or broadcaster to dynamically select theshape, size, or position of the region.

Another method of distinguishing the region of interest is toautomatically compare adjacent frames and select a subset region thatcontains the largest change. For example, analysis of a video of asprinter would show that the leg and arm positions of the athlete arethe biggest change. The torso, and perhaps the head, may remain fairlystable. An automatic analysis of adjacent video frames would detect thatthe regions containing the legs and arms were of greatest change. Thoseareas could be automatically selected as the region of greatest interestfor applying the methods of the present invention.

In some cases, the area of interest is encoded and compressed such thatimage quality remains high and is transmitted and displayed morefrequently. The unselected area, which is important for reference andorientation, is encoded, compressed, and transmitted at a lower quality,resolution, and frame rate. The viewer or broadcaster can change thearea of focus to get more detail if the area outside the current area offocus becomes interesting. If a frame has to be dropped to maintaintransmission frame rate, the plane of less quality will be droppedbefore frames from the more important plane, thus allowing for morebandwidth for the selected area of interest.

In some cases, a video image will contain a marker that can easily bedetected by analysis of the video. For example, in a baseball game,generally, any area of the image surrounded by grass or dirt is moreimportant that the sky or stadium. The grass and dirt can be detected asa marker and the regions substantially enclosed by those markers, namelythe ball, bat, and players, can be automatically distinguished as theregions of greater interest. Also, for example, in a weather map videobeing broadcast to the San Francisco Bay Area audience, the region ofthe map corresponding to Northern California can easily be detected andthat portion of the video can automatically be distinguished as theregion of greater interest.

Objects and Advantages

Accordingly, beside the objects and advantages of the method describedabove, some additional objects and advantages of the present inventionare:

-   -   (a) to provide efficient processing of image data prior to        display on a computer display.    -   (b) to provide efficient processing of image data being captured        in real time with a video digitizer.    -   (c) to reduce the time necessary to process the image data.    -   (d) to provide a method of compressing and decompressing video        signals so that the video information can be transported across        a digital communications channel in real time.    -   (e) to provide a method of compressing and decompressing video        signals such that compression can be accomplished with software        on commercially available computers without the need for        additional hardware for either compression or decompression.    -   (f) to provide a high quality video image without the blocking        and smearing defects associated with prior art lossy methods.    -   (g) to provide a high quality video image that suitable for use        in medical applications.    -   (h) to enhance images by filtering noise or recording artifacts.    -   (i) to provide a method of compression of video signals such        that the compressed representation of the video signals is        substantially reduced in size for storage on a storage medium.    -   (j) to provide a level of encryption so that images are not        directly viewable from the data as contained in the        transmission.    -   (k) to provide efficient encoding of Doppler enhanced images.    -   (l) to reduce the size of an encoded data buffer that contains        Doppler enhancement.    -   (m) to provide efficient encoding for video images that contain        distinguishable regions.    -   (n) to reduce the size of an encoded data representing a video        stream.    -   (o) to reduce the bandwidth required to transmit an encoded        video stream.    -   (p) to provide user control of the region selection and        compression methods.    -   (q) to automatically detect selection regions by detecting        viewer eye movements.    -   (r) to automatically detect selection regions by comparing areas        of change.    -   (s) to automatically detect selection regions by detecting        markers in the image.

DRAWING FIGURES

In the drawings, closely related figures have the same number butdifferent alphabetic suffixes.

FIG. 1A shows a basic computer architecture.

FIG. 1B shows components for video digitizing.

FIG. 1C shows components for computer display.

FIG. 1D shows a multi-level cache architecture.

FIG. 1E shows a DMA architecture.

FIG. 1F shows images copied between an I/O video RAM and main memory.

FIG. 1G shows an input image being input and encoded.

FIG. 1H shows encoded data being decoded and output.

FIG. 1I shows the high level steps of compression and decompression ofan image.

FIG. 2A to 2H show alternatives for selecting a pixel value forencoding.

FIG. 2A shows the four channel format of a pixel.

FIG. 2B shows the three color components of a pixel.

FIG. 2G shows selection of a component of a grayscale pixel.

FIG. 2F shows a selection of the red component of a red scale pixel.

FIG. 3A shows the basic format of the ZLN variable encoding format

FIG. 3B shows an example of a code where N is 5 bits wide and U is 3bits wide.

FIG. 4A shows the flowchart for the compression method.

FIG. 4B shows an image and a corresponding stream of pixels.

FIG. 4C shows row by row copy of a subset image.

FIG. 4D shows a flowchart for copying a subset image from I/O RAM to amemory buffer.

FIG. 4E shows a flowchart for copying a memory buffer to a subset imagein I/O RAM.

FIG. 5A to 5C shows the formats for the run-length encoding of the RHNmethod.

FIG. 5D shows the ZLD format.

FIG. 5E shows the three formats of the three scales.

FIG. 5F shows an alternate embodiment of the grayscale format.

FIG. 5G shows the ZLD format embedded in the ZLN format.

FIG. 6 shows a series of codes and the resulting encoded stream.

FIG. 7 shows a series of codes and the resulting encoded stream of theRHN method.

FIG. 8A shows examples of variable formats.

FIG. 8B shows a format that preserves 9 bits of color.

FIG. 9 shows the flow chart for the decompression method.

FIG. 10 shows image stretching by interpolation.

FIG. 11A shows an example encode table.

FIG. 11B shows a corresponding grayscale decode table.

FIG. 11C shows a corresponding red scale decode table.

FIGS. 12A and 12B show machines for compressing and decompressing,respectively.

FIG. 12C shows a compressor and decompressor connected to a storagemedium.

FIG. 12D shows a compressor and decompressor connected to acommunications channel.

FIG. 13A shows elements of a compressor.

FIG. 13B shows an embodiment of an encoding circuit.

FIG. 14 shows a generic pixel sub-sampler.

FIG. 15 shows elements of a decompressor.

FIG. 16A shows elements for setting width, height, frame rate,brightness, and contrast which are variably altered by a receiver.

FIG. 16B shows elements for setting the number of pixel bits that arevariably altered by a receiver.

FIG. 17 shows a lossless compression step for further compression of anencoded data buffer.

FIG. 18 shows images being enlarged by stretching.

FIG. 19A shows an example of image separation.

FIG. 19B shows compression and decompression.

FIG. 20 shows a flow chart of the separation and compression method.

FIG. 21 shows different rates for separated image streams.

FIG. 22A shows a system for user selection of separation regions.

FIG. 22B through 22E show examples of various selection shapes.

REFERENCE NUMERALS IN DRAWINGS

 100 compression steps  101 CPU  102 output  103 memory  104 input  105video source  110 sub-sampling step  111 video digitizer  113 capturevideo RAM  120 display video RAM  121 video display  122 I/O RAM  123cache  124 CPU cache  125 DMA circuitry  130 encoding step  140 encodeddata  150 decompression steps  160 decoding step  180 imagereconstitution step  190 buffer  192 buffer-image copy  194 image  196encoder  197 decoder  198 encoded data  200 32 bit pixel value (fourchannel format)  202 blue channel  204 green channel  206 red channel 208 alpha channel  210 24 bit pixel value  212 blue component  214green component  216 red component  220 RGB averaging diagram  222 bluevalue  224 green value  226 red value  228 averaged value  230 blueselection diagram  232 blue instance  234 green instance  236 redinstance  240 selected blue value  250 green selection diagram  252 DMAcontrol  254 DMA-Memory bus  256 DMA-I/O bus  260 selected green value 270 red selection diagram  280 selected red value  290 grayscale pixel 292 grayscale blue  294 grayscale green  296 grayscale red  298selected grayscale value  299 filtered pixel value  300 N  301 U  302 W 310 pixel bit 7  312 pixel bit 6  314 pixel bit 5  316 pixel bit 4  318pixel bit 3  320 pixel bit 2  322 pixel bit 1  324 pixel bit 0  325 8bit pixel  330 5 bit sample  332 sample bit 4  334 sample bit 3  336sample bit 2  338 sample bit 1  340 sample bit 0  350 3 low order bits 360 formatted code  362 encoded bit 4  364 encoded bit 3  366 encodedbit 2  368 encoded bit 1  370 encoded bit 0  380 3 bit count value  400encode flowchart  402 encode entry  403 encode initialization step  404get pixel step  405 get value step  406 lookup encoded value step  408compare previous  410 increment counter step  412 check count overflow 414 new code step  416 check end of data  418 set done  420 counteroverflow step  422 check done  428 encode exit  430 image  431 firstimage line  432 second image line  434 last image line  436 first bufferline  438 second buffer line  439 last buffer line  440 image width  442image copy start  444 image copy initialization step  446 set counterstep  448 image copy done decision  450 image height  452 image copystep  454 update pointers step  456 increment index step  458 image copyexit  460 pixel stream  462 buffer copy start  464 buffer copyinitialization step  466 set counter step  468 buffer copy done decision 470 buffer copy step  472 update pointers step  474 increment indexstep  476 buffer copy exit  490 super-image  500 code byte  501 S1  502S0  503 S  505 E  510 flag bit  511 Doppler/grayscale flag zero  512don't care  514 grayscale value  515 grayscale code  520 repeat code 521 Doppler/grayscale flag one  522 blue/red flag zero  524 red scalevalue  525 red Doppler scale code  530 count  531 secondDoppler/grayscale flag one  532 blue/red flag one  534 blue scale value 535 blue Doppler scale code  541 second Doppler/grayscale flag zero 544 extended value  545 extended grayscale code  550 data code  551 S1in ZLN format  552 S0 in ZLN format  553 S in ZLN format  554 E in ZLNformat  555 C  557 X  560 wasted bits  565 data bit 6  570 data bit 5 575 data bit 4  580 data bit 3  585 data bit 2  590 data bit 1  595data bit 0  610 decimal values  620 first value  622 second value  624third value  626 fourth value  628 fifth value  630 sixth value  632seventh value  640 binary code  650 first byte  651 first data  652first count  653 second byte  654 second data  655 second count  656third byte  657 third data  658 third count  740 RHN binary code  803ZL3 format  804 ZL4 format  805 ZL5 format  808 ZL8 format  809 ZL9format  812 ZL12 format  820 ZL9C format  900 decode entry  901 decodeinitialize step  902 get code step  908 decode lookup step  909 checkzero count  910 place pixel step  914 reset counter step  916 checklength  918 decode exit  920 decode flowchart 1010 first adjacent pixel1012 second adjacent pixel 1014 first subsequent adjacent pixel 1016second subsequent adjacent pixel 1052 interpolated pixel 1054interpolated pixel 1056 interpolated pixel 1058 interpolated pixel 1060interpolated pixel 1100 encryption table 1110 decryption table 1112grayscale red bits 1114 grayscale green bits 1116 grayscale blue bits1120 red scale decode table 1122 red scale red bits 1124 red scale greenbits 1126 red scale blue bits 1200 video frames 1205a first video frame1205b second video frame 1205n nth video frame 1210 compressor 1215video signal 1220 series of encoded data 1225a first encoded data 1225bsecond encoded data 1225n nth encoded data 1225 encoded data buffer1230a first received encoded data 1230b second received encoded data1230n nth received encoded data 1230 received encoded data 1235 encodeddata stream 1238 received encoded data (series) 1240 I/O device 1245input encoded data stream 1250 decompressor 1260a first decoded videoframe 1260b second decoded video frame 1260n nth decoded video frame1260 decoded video frame 1268 decoded video frames 1270 video sequence1280 storage medium 1290 communications channel 1310 video digitizer1320 path 1320 1330 video memory 1331 scan 1332 pixel index 1340 path1340 1350 encoding circuit 1360 path 1360 1370 encoded data 1380a 24 to5 bit sub-sampler 1380b 24-bit RGB to 5 bit sub-sampler 1380c 32-bit RGBto 5 bit sub-sampler 1380d color 9-bit sub-sampler 1380e YUV sub-sampler1380f 36-bit RGB to 24-bit sub-sampler 1380g 15-bit sub-sampler 1380pixel sub-sampler 1382 pixel extractor 1383 value path 1384 coder 1385path 1385 1390 data/count 1392 code index 1395 path 1395 1400 24-bit tovariable bit sub-sampler 1401 generic 3-bit sub-sampler 1402 generic4-bit sub-sampler 1403 generic 8-bit sub-sampler 1404 generic 10-bitsub-sampler 1410 number of bits selector 1420 number of bits indicator1430 36-bit to variable bit sub-sampler 1440 24/36 bit variable bitsub-sampler 1450 second selector 1460 selection logic 1470 selectionsignal 1510 decoding circuit 1520 decoded pixel values 1530 decoderpixel index 1540 image memory 1600 transmitter 1610 receiver 1615setting control path 1620 frame sub-sampler 1621 path 1621 1630 selectedframe 1632 pixel from frame 1640 transmitter pixel sub-sampler 1642 path1642 1650 run length encoder 1660 settings 1661 brightness 1662 contrast1663 height 1664 width 1665 frame rate 1670 frame selector 1675 frameselect indicator 1680 number of pixel bits setting 1690 alternatetransmitter 1700 run-length encoding step 1710 run-length encoded output1720 further lossless compression step 1730 further lossless compressionoutput 1800 unstretched frame 1810 enlarged image 1820 stretching step1900 original image 1902 grayscale region 1904 grayscale triangle 1910red oval 1912 blue circle 1915 first path 1920 first plane 1925 secondpath 1930 second plane 1940 first compression method 1945 secondcompression method 1950 first encoded data 1955 second encoded data 1960composite buffer 1970 first decompression method 1975 seconddecompression method 1980 first decoded image 1985 second decoded image1990 copy path 1995 overlay path 1999 combined image 2000 entry point2002 path 202 2004 get pixel step 2006 path 206 2008 which planedecision 2010 path 210 2012 add to first buffer step 2014 path 214 2016put mask in second buffer step 2018 path 218 2020 path 220 2022 add tosecond buffer step 2024 path 224 2026 put mask in first buffer step 2028path 228 2030 increment buffer pointers step 2032 path 232 2040 donedecision 2050 path 250 2060 path 260 2062 compress first buffer step2064 path 264 2066 compress second buffer step 2068 path 268 2070 exitpoint 2100 first transfer 2101 second original image 2102 second firstplane 2103 second second plane 2105 second first decoded image 2106reuse of second decoded image 2107 second combined image 2110 secondtransfer 2111 third original image 2112 third first plane 2113 thirdsecond plane 2115 third first decoded image 2116 third second decodedimage 2117 third combined image 2120 third transfer 2121 fourth originalimage 2122 fourth first plane 2123 fourth second plane 2125 fourth firstdecoded image 2126 reuse of third decoded image 2127 fourth combinedimage 2130 fourth transfer 2140 fifth transfer 2150 sixth transfer 2180input video stream 2190 output video stream 2200 user input 2205 userinput path 2210 display generator 2215 image source path 2220 imagesource 2225 display path 2230 display 2235 control data path 2240control data 2250 bust region 2260 background region 2270 oval region2280 outer region 2282 inner region 2284 hair region 2285 face region2286 coat region 2287 shirt region 2288 tie region 2290 rectangularregion 2299 combined region h image height w image width x imagehorizontal offset y image vertical offset

SPECIAL DEFINITIONS

buffer a special area in main memory used to hold data temporarily forprocessing by a CPU

I/O RAM—a random access memory which is associated with an I/O device,and which is distinct from main memory

mask value—a uniform pixel value, typically black (all zeros) or white(all ones), used to indicated that the position in the pixel map is tobe processed differently.

mask—a region comprising a collection of mask values.

memory—main memory which is distinct from an I/O RAM, a CPU cache, or anexternal cache

memory copy function—a computer program that copies a block data betweena memory address and another memory address

plane—a image containing pixels selected from the original image basedon their distinguishing characteristics and containing mask pixelsholding the place of unselected pixels.

DESCRIPTION OF THE INVENTION

FIG. 1A to 1C—Computer Architectures

FIG. 1A is a block diagram showing the basic components of a computer,comprising an input 104, a CPU 101, an output 102, and memory 103.

FIG. 1B shows an embodiment of the computer input 104 specialized toinput video data. A video source 105 is connected to a video digitizer111. The video digitizer 111 converts the analog video signal from thevideo source 105 to a digital format. Some video digitizers transfer thevideo data to memory 103 for storage. Alternatively, some videodigitizers contain capture video RAM 113 which can store the capturedvideo data on the video digitizer 111 hardware without using memory 103for storage.

FIG. 1C shows an embodiment of the computer output 102 specialized tooutput video data. A video display 121 (also knows as a computermonitor) displays graphical information based on data contained in adisplay video RAM 120. Programs running on the CPU 101 determine thecontents of the display video RAM that is then shown by pixels on thevideo display 121.

FIGS. 1D and 1E—Caching and DMA

FIG. 1D is a block diagram showing computer architecture where thereoptionally are two levels of caches. The CPU 101 has an internal cacheknown as a CPU cache 124 that can store copies of recently accessedmemory blocks that contain program instructions or data. A cache 123stores copies of recently accessed blocks of memory data, because thecache 123 is outside the processor it is sometimes referred to as anexternal cache. In an architecture where the CPU has an internal CPUcache 124, the cache 123 may also be referred to as a level 2 cache.

If a copy of a block of memory data is in the CPU cache 124 or thememory cache 123, the CPU 101 can access it much faster than if the datahas to be fetched from memory 103. If the data is not available to theCPU 101, the CPU 101 stalls causing there to be cycles where no usefulprocessing is being done. The use of caches (123, 124) can have asignificant impact of the speed of data processing.

It is common for input and output device registers and memories to bemapped into the memory address range. This is called memory mapped I/O.In a computer architecture that uses memory mapped I/O, the randomaccess memory (RAM) associated with computer input 100 and output 102devices can be accessed by programs running on the CPU as if they werememory 103 RAM. Because the I/O RAM 122 can be modified by itsrespective input 104 or output 102 device, special provisions are madeso that the blocks of memory from I/O RAM 122 are not stored in thecache 123 or the CPU cache 124 (or if they are stored in the cache theyare marked as invalid so that the CPU will fetch the current contents ofthe I/O RAM 122 rather than use the obsolete data in the cache).Examples of I/O RAM 122 include capture video RAM 113 and display videoRAM 120.

FIG. 1E shows a computer architecture with direct memory access (DMA)circuitry 125. Without DMA circuitry 125, the CPU 101 must be involvedin transferring data from memory 103 to I/O RAM 122. This CPUinvolvement takes the CPU 101 processing power away from executing otherprogram instructions and adds overhead to handle the interruptions. DMAcircuitry 125 is used to copy blocks of data directly between memory 103and I/O RAM 122. A DMA operation is initiated by the CPU 101 with a DMAcontrol 252 sent from the CPU 103 to the DMA circuitry 125. Once the DMAoperation is initiated the CPU can return to other work. The DMAcircuitry moves the data from memory 103 to I/O RAM 122 along theDMA-memory bus 254 and DMA-I/O bus 256 or from I/O RAM 122 to memory103. In practice, the DMA circuitry may become a secondary bus master ofa system bus that interconnects the CPU 101, I/O RAM 122, and memory103. Once the data transfer is complete the DMA circuitry 125 notifiesthe CPU.

Processing Speed Improvement—FIG. 1F to 1H

When video data is being displayed or captured the storage (memory 103or I/O RAM 122) holding the data is continually being accessed by thevideo display circuitry or video digitizing circuitry. Also the capturevideo RAM 113 and the display video RAM 120 typically is not cached by aCPU 101 in any cache (123 or 124), so when processing the video data forcompression, encryption, enhancement, or decompression it issignificantly faster to process the data in cacheable main memory.

The present invention uses a memory copy function (similar to a memcpyfunction or a substantially similar set of computer instructions) tocopy the desired image data from an I/O RAM 122 to a cacheable mainmemory 103 (FIG. 1D) where it can be more efficiently processed. Afterthe processing is done, the processed image is then copied back to thedisplay video RAM 120 for display on the video display 121 (FIG. 1C).

FIG. 1F shows a buffer 190 in memory 103 and an image 194 stored in I/ORAM 122. The buffer-image copy 192 of data between the buffer 190 andthe image 194 is shown as bi-directional arrows. Once the image data iscopied from the image 194 to the memory buffer 190 it can be much moreefficiently processed by the CPU 103. FIG. 1G shows an encoder 196program which accesses the buffer 190 applying enhancement, compression,or encryption algorithms as needed to produce encoded data 198. Theencoded data 198 can be stored on a storage device or transferred over anetwork to another computer. FIG. 1H shows a decoder 197 programprocessing the encoded data 198 into another instance of a memory buffer190. The decoder can decrypt, decompress, or enhance the encoded data asneeded and place the resulting data in a memory buffer 190.

This invention discovered that is was much more efficient to write thedecoded data to a memory buffer 190 instead of writing it directly toimage 194 in I/O RAM 122 as each pixel is processed. Once the decoderprocessing is complete, the buffer-image copy 192 is used to transferthe data from the buffer 190 to the I/O RAM 122. The I/O RAM could be adisplay video RAM 120 as shown in FIG. 1C.

Not Obvious

The speed improvement yielded by this invention was not obvious to oneskilled in the art of computer programming. The video data is large, upto 1.2 million bytes, and the time to copy it from one buffer to anothergenerally is thought to be overhead that will decrease performance. Thisinvention teaches that because of hardware lockout, collisions with thevideo circuitry, the lack of data caching in the CPU cache 124 or memorycache 123, or other factors, the extra copy can significantly reduce theprocessing time, and thus reduce the overall time required to processthe data and to display or capture the video data.

The memory copy routine used in the buffer-image copy 192 may useprocessor specific code, or other methods, to move blocks of databetween the memory 103 (or the caches (123, 124) and the I/O RAM 122.

The methods of this invention are much more efficient (due to I/O RAMlockouts and conflicts) than processing each pixel a byte or word at atime in place in I/O RAM 122.

Alternatively, DMA circuitry 125 (FIG. 1E) may be used to increase thespeed of transfer between memory 103 and the I/O RAM 122.

In one embodiment of this invention the entire image is copied by asingle call to the memcpy function. This has the advantage of onlymaking one function call.

FIG. 1I—Compression and Decompression Steps

FIG. 1I illustrates a sequence of compression steps 100 and a sequenceof decompression steps 150 of the present invention. The compressionsteps 100 comprise a sub-sampling step 110 and an encoding step 130.After completion of the compression steps 100, a stream of encoded data140 is output to either a storage medium or a transmission channel. Thedecompression steps 150 comprise a decoding step 160 wherein the streamof encoded data 140 is processed and an image reconstitution step 180.

FIGS. 2A to 2H-Selecting Pixel Values for Encoding

FIGS. 2A to 2G illustrate alternatives for selecting a pixel value forencoding. The sub-sampling step 110 (FIG. 1) includes sub-sampling of apixel value to obtain a variable selected number of bits.

Video digitizing hardware typical has the options of storing the pixelvalues as a 32 bit pixel value 200 or a 24 bit pixel value 210, shown inFIG. 2A and FIG. 2B, respectively. The 32 bit pixel value 200 iscomposed of a blue channel 202, a green channel 204, a red channel 206,and an alpha channel 208. Each channel contains 8 bits and can represent256 saturation levels for the particular color channel. For each channelthe saturation intensity value of zero represents the fully off state,and the saturation intensity value of “255” represents the fully onstate. A common alternative not shown is a sixteen-bit format where thethree color channels contain 5 bits each and the alpha channel is asingle bit. The present invention anticipates the use of the colorchannels of 16 bit pixel value is a manner substantially the same as the32-bit pixel value 200 except the number of bits per channel is 5instead of 8.

The 24-bit pixel value 210 is composed of a blue component 212, a greencomponent 214, and a red component 216. There is no component for thealpha channel in the 24 bit pixel value 210. Regardless of thestructure, the blue channel 202 is equivalent to the blue component 212,the green channel 204 is equivalent to the green component 214, and thered channel 206 is equivalent to the red component 216.

In the present invention, the 32 bit pixel value 200 alternative ispreferred due to the consistent alignment of 32 bit values in mostcomputer memories; however for simplicity of illustration the alphachannel 208 will be omitted in FIGS. 2C to 2G.

If the video signal is digitized in color, the three color componentsmay have different values. For example in FIG. 2C, a RGB averagingdiagram 220 illustrates a blue value 222 of 35 decimal, a green value224 of 15, and a red value 226 of 10. One alternative is to sub samplefrom 24 bits to 8 bits by averaging the three color values to obtain anaveraged value 228 that, in this example, has the value of 20:(10+15+35)/3=20. This will produce a grayscale image. Alternatively, acolor image can be preserved by sampling bits from each color component(see FIG. 8B).

FIG. 2D illustrates another alternative for selecting an 8 bit value ina blue selection diagram 230. In this example, a blue instance 232 hasthe value of 35, a green instance 234 has the value of 15, and a redinstance 236 has the value of 10. In this alternative the blue instance232 is always selected as a selected blue value 240.

FIG. 2E illustrates another alternative for selecting an 8 bit value ina green selection diagram 250. In this alternative the green instance234 is always selected as a selected green value 260.

FIG. 2F illustrates another alternative for selecting an 8 bit value ina red selection diagram 270. In this alternative the red instance 236 isalways selected as a selected red value 280.

If the video signal being digitized is grayscale, the three colorcomponents will have the same values. For example in FIG. 2G, agrayscale pixel 290 comprises a grayscale blue 292 with a value ofdecimal 40, a grayscale green 294 with a value of 40, and a grayscalered 296 with a value of 40. Because the values are all the same, itmakes no difference which grayscale color component is selected, aselected grayscale value 298 will have the value of 40 in this example.

The preferred embodiment of this invention uses the low order byte ofthe pixel value, which is typically the blue component as shown in FIG.2D.

FIG. 2H illustrates a filtered pixel value 299 of 8 bits that may beselected by one of the alternatives described above. In these examples,the filtered pixel value 299 is equivalent to items referenced bynumerals 228, 240, 260, 280, or 298. This reduction of the 32 bit pixelvalue 200 or the 24 bit pixel value 210 contributes a reduction in datasize of 4:1 or 3:1, respectively. This reduction recognizes that forsome images, such as medical images or grayscale images, no relevantinformation is lost.

For additional compression, the filtered pixel value 299 can variablyselect any number of bits. For example, selection of the mostsignificant four bits instead of all eight bits filters noise that mayshow up in the low order bits may be very suitable for an image such asone produced by an ultrasound medical device. An example of this isshown by ZL4 804 in FIG. 8A.

FIGS. 3A and 3B-Encoding Formats

Speed of compression and decompression may be enhanced if the algorithmsfit into computer memory native storage elements such as 8 bit bytes, 16bit words, or 32 bit double words, or some other size for which thecomputer architecture is optimized.

A grayscale image may be stored at a higher bit level than the actualvalues require. This may occur when an image is generated by an imagingtechnology such as radar, ultrasound, x-ray, magnetic resonance, orsimilar electronic technology. For example an ultrasound machine mayonly produce 16 levels of grayscale, requiring 4 bits of data per pixel,but the image digitizing may be performed at 8 to 12 bits per pixel. Inthis example, the low order bits (4 to 8) respectively provide nosignificant image data.

In the present invention, a fast and efficient compression and encodingmethod is implemented by using unused bits to store a repeat count forrepeated values.

The most significant N bits of the pixel value are selected where N 300is the number of significant bits (determined by data analysis or byuser selection). If N 300 is less than W 302, where W is a nativemachine data type such as 8 bit byte, 16 bit word, or 32 bit double wordor some other size for which the computer architecture is optimized,then W-N equals the number of unneeded bits, U 300. A repeat count, C,can contain a value from 1 to CMAX where CMAX is 2 to the power of U.For example, if U equals 4, C can be a number from 1 to 16. In practicethe maximum value will be encoded as a zero because the high order bitis truncated. In the example, decimal 16 has a binary value “10000” willbe stored as “0000”.

For example, when W is 8, value pairs for N and U could include withoutlimitation (2,6), (3,5), (4,4), (5,3), and (6,2). When W is 16, valuepairs for N and U could include without limitation (2,14), (3,13),(4,12), (5,11), (6,10), (7, 9), (8, 8), (9, 7), (10, 6), (11, 5), (12,4), (13, 3), and (14, 2). When W is 32, value pairs for N and U couldinclude without limitation all combinations of values pairs for N and Uwhere N+U equals 32 and N>1 and U>1. When W is not a multiple of 8,value pairs for N and U could include without limitation allcombinations of values pairs for N and U where N+U equals W and N>1 andU>1.

FIG. 3A shows the encoded format where N 300 represent the N mostsignificant bits of the pixel value 299, U 301 represents the bits thatare not used for the data and are used for the repeat count, and W 302where W is the width of the encoded data and equal to sum of N and U. Asstated above W is preferably a native machine element.

FIG. 3B illustrates bit sub-sampling where N's 300 bit width is 5, U's301 bit width is 3, and W 302 is 8. The high order 5 bits 310-318 of an8 bit pixel 325 are extracted to form a five bit sample 330. The lower 3bits of 330 are ignored bits 350. In the formatted code 360, the ignoredbits 350 are replaced with the repeat count value 380.

Encoding

The most significant N bits of each pixel are selected from the image toobtain value V.

In the encryption embodiment of this invention V may be used to selectan encoded value, E, from the encoding table. E is also a N-bit value.The number of elements in the encode table 1100 (FIG. 1I, shown as anencryption table), is 2 to the Nth power.

In the other embodiments of this invention V is used as E.

E is saved as the prior value, P. For each subsequent pixel, the encodedvalue, E, is obtained and compared to the prior value, P. If the priorvalue, P, is the same as E, then a repeat counter, C, is incremented;otherwise the accumulated repeat count, C, for the prior value, P, ismerged with P and placed in an array A that implements the encoded data140 (FIG. 1) buffer. For example, if W is 8 and N is 4 and C is 10, U is4, CMAX is 16, and ((P<<U)|C) is the merged value. If the repeat count,C, is greater CMAX, then CMAX is merged with P ((P<<U)|CMAX) and placedin the encoded data 140 (FIG. 1) buffer, A. CMAX is subtracted from Cand merged values are placed in A until C is less than CMAX. All pixelsare processed in this manner until the final value is compressed andencoded. The length, L, of the encoded data 140 (FIG. 1) is also placedin the encoded data 140 buffer.

FIG. 4A—Encode Flowchart

FIG. 4A illustrates the encode flowchart 400 which represents thedetails of the encryption embodiment of the encoding step 130 (FIG. 1)for the present invention.

The encoding begins at an encode entry 402. In an encode initializationstep 403, a prior value P is set to a known value, preferably decimal“255” or hexadecimal 0xFF, a repeat counter C is set to zero, an encodedlength L is set to 0, and a completion flag “Done” is set to a logicalvalue of false. Next, a get pixel step 404 obtains a pixel from theimage being encoded. At a get value step 405, a value V is set to the Nbit filtered pixel value 299 as derived from the pixel using one of themethods shown in FIG. 2C to 2G, preferably the fastest as explainedabove, and extracting the N most significant bits. At a lookup encodedvalue step 406, an encoded value E is set to the value of one of thecodes 1105 (FIG. 11A) of the encode table 1100 as indexed by V. (In thenon-encrypted embodiment of this invention, step 406 is bypassed becauseV is used as E) Next, a “compare previous” 408 decision is made bycomparing the values of E and P. If the values are the same, anincrement counter step 410 is executed and flow continues to the getpixel step 404 that obtains the next pixel from the image.

If the encode value E does not match the prior value P, then a checkcount overflow 412 decision is made. If the counter C is less than orequal to CMAX, then a new code step 414 is executed, otherwise a counteroverflow step 420 is executed.

At step 414, the counter C is masked and bit-wise OR-ed with P shiftedleft by U bit positions and is placed in the A at the next availablelocation as indexed by the encoded length L. Then, continuing insideflowchart step 414, L is incremented, the repeat count C is set to 1 andthe prior value P is set to E. After step 414, a “check end of data”decision is made by checking to see if there are any more pixels in theimage, and, if not, if the last value has been processed. Because thismethod utilizes a read ahead technique step 414 must be executed onemore time after the end of data is reached to process the lastrun-length. If there is more data in the image, flow continues to acheck of the completion flag “Done” at step 422. If the check indicatesthat the process is not completed, flow continues to step 404.

If the end of data is reached but the completion flag “Done” is stillfalse, flow continues to a set done step 418. At step 418, thecompletion flag “Done” is set to logical true, and flow continues todecision 412 where the last run-length will be output and flow willeventually exit through step 414, decision 416, decision 422, and thenterminate at encode exit 428.

It is possible for the repeat count C to become larger than CMAXrequiring more bits than allocated by this method. This situation ishandled by making the “check count overflow” 412 decision and executingthe “counter overflow” step 420. At step 420, the counter C is maskedand bit-wise OR-ed with P shifted left by U bit positions and is placedin the A at the next available location as indexed by the encoded lengthL. Then, continuing inside flowchart step 414, L is incremented, and therepeat count C is decrement by CMAX. After step 420, flow continues tothe “check count overflow” 412 decision. Thus when the encode value Erepeats more than CMAX times, multiple sets of repeat counts and encodedvalues are output to the encoded data 140 buffer.

This entire process is repeated for each image or video frame selectedduring optional image sub-sampling (see 110 in FIG. 1) and the encodedlength L is transmitted with the encoded data associated with eachframe. The encoded length varies from frame to frame depending on thecontent of the image being encoded.

FIG. 4B—Image and Pixel Stream

FIG. 4B illustrates an image and its corresponding stream of pixels. Arectangular image 430 is composed of rows and columns of pixels. Theimage 430 has a width 440 and a height 450, both measured in pixels. Inthis illustrative embodiment, pixels in a row are accessed from left toright. Rows are accessed from top to bottom. Some pixels in the imageare labeled from A to Z. Pixel A is the first pixel and pixel Z is thelast pixel. Scanning left to right and top to bottom will produce apixel stream 460. In the pixel stream 460, pixels A and B are adjacent.Also pixels N and O are adjacent even though they appear on differentrows in the image. If adjacent pixels have the same code the process inFIG. 4A will consider them in the same run.

Because the video signal being digitized is analog there will be someloss of information in the analog to digital conversion. The videodigitizing hardware can be configured to sample the analog data into theimage 430 with almost any width 440 and any height 450. The presentinvention achieves most of its effective compression by sub-sampling thedata image with the width 440 value less than the conventional 640 andthe height 450 value less than the convention 480. In a preferredembodiment of the invention, for use in a medical application with T1Internet transmission bandwidth, image dimensions are sub-sampled at 320by 240. However an image dimension sub-sampling resolution of 80 by 60may be suitable for some video application.

FIG. 4C—Preferred Embodiment

In a preferred embodiment, only a subset image 194 of the data in I/ORAM 122 is of interest for processing, so the memory copy function iscalled repeatedly to copy each line of desired image data. For exampleif the desired subset is 196 by 124, the memory copy function is called240 times and copies 320 pixels each time. This has the advantage ofonly copying the desired data. Even though there is more overhead indetermining how to copy the subset and in calling the memory copyfunction multiple time, the time saved by copying less data more thancompensates for the additional overhead. Less memory is used to hold themain memory buffer and less data must be processed.

FIG. 4C is a diagram of the buffer 190 and the image 194 that shows moredetail than FIG. 1F. The subset image 194 is contained within asuper-image 490. When a television video signal is digitized there areportions of the signal that are not visible on most television displays.The video digitizer often will process all of the video signal producinga super-image 490 that contains data that surrounds the subset image 194and the surround data typically is of no interest. If the origin of thesuper-image 490 is (0,0) the image 194 of interest can be found at acoordinate (x, y) composed of the image horizontal offset x and theimage vertical offset y. The image width w and the image height can beused to allocate the memory buffer 190, rather than copying the entiresuper-image 490. The coordinate of the last pixel of the desired imageis (x+w, y+h).

In a preferred embodiment, the first image line 431 (starting at (x,y))is copied (192) to the first buffer line 430 for the length of the imagewidth w. Next the second image line 432 is copied to the second bufferline 438. Each line is copied until the last image line 434 is copied tothe last buffer line 439. After the desired data is copied in thismanner the buffer 190 can be efficiently processed. Buffer 190 issmaller than the super-image 490 and the data of interest is contiguousso it can be processed more efficiently. Buffer 190 can be cached andwill have typically no conflict from other accesses.

FIG. 4C also illustrates the reverse process of copying a buffer 190containing processed data to a super image 490 in an I/O RAM 122 (FIG.1F). Each line of the buffer 190 is copied (192) to the image 194 in thesuper image 490 at the desired offset (x,y). In this reverse process thefirst buffer line 436 is copied to the first image line 431. The secondbuffer line 438 is copied to the second image line 431, and so forth,until the last buffer line 439 is copied to the last image line 434. Thesame advantages of buffer 190 being smaller, contiguous, cacheable, andconflict free also apply to the reverse process.

FIG. 4D—Image Copy Flowchart

FIG. 4D is a flow chart for the method of copying the image 194 tobuffer 190 as shown in FIG. 4C. The method starts at an image copy start442 entry point. Next an image copy initialization step 444 comprisingthe following is executed: the line size is set to the image width w.

-   -   the number of lines is set to the image height h.    -   the row size is calculated by dividing the total bytes in a row        of the super image by the number of bytes per pixel.    -   the copy size is calculated by multiplying the line size by the        number of bytes per pixel.    -   the source pointer is set the base address of the image 490 plus        the calculation of the number of bytes to get to the (x,y)        offset: ((y*row size+x)*bytes per pixel).    -   the destination pointer is set to the base address of the buffer        190.        Next, in a set counter step 446, the row index is set to 0. An        image copy done decision 448 is made by comparing the row index        to the number of lines. If one or more lines still need to be        copied, flow continues to an image copy step 452. In the image        copy step 452, the memory copy function is called to copy        copy-size bytes from the current source pointer to the current        destination pointer (effectively copying a line of the image 194        to the buffer 190). Next, in an update pointers step 454, the        source pointer is incremented by the number of bytes in a row of        the super image (effectively addressing the beginning of the        next line of the image 194), and the destination pointer is        incremented by the number of bytes in a line of the buffer 190        (effectively addressing the beginning of the next line of the        buffer 190). Next in an increment index step 456, the row index        is increment. Flow continues to the image copy done decision        448, and the loop continues until each line of the image 194 is        copied. When the image has been fully copied, flow terminates at        an image copy exit 458 point.        FIG. 4E—Buffer Copy Flowchart

FIG. 4E is a flow chart for the method of copying the buffer 190 to theimage 194 as shown in FIG. 4C. The method starts at a buffer copy start462 entry point. Next a buffer copy initialization step 464 comprisingthe following is executed:

-   -   the line size is set to the image width w.    -   the number of lines is set to the image height h.    -   the row size is calculated by dividing the total bytes in a row        of the super image by the number of bytes per pixel.    -   the copy size is calculated by multiplying the line size by the        number of bytes per pixel.    -   the destination pointer is set the base address of the super        image 490 plus the calculation of the number of bytes to get to        the (x,y) offset: ((y*row size+x)*bytes per pixel).    -   the source pointer is set to the base address of the buffer 190.        Next, in a set counter step 466, the row index is set to 0. A        buffer copy done decision 468 is made by comparing the row index        to the number of lines. If one or more lines still need to be        copied, flow continues to a buffer copy step 470. In the buffer        copy step 470, the memory copy function is called to copy        copy-size bytes from the current source pointer to the current        destination pointer (effectively copying a line of the buffer        190 to the image 194). Next in an update pointers step 472, the        destination pointer is incremented by the number of bytes in a        row of the super image (effectively addressing the beginning of        the next line of the image 194), and the source pointer is        incremented by the number of bytes in a line of the buffer 190        (effectively addressing the beginning of the next line of the        buffer 190). Next in an increment index step 474, the row index        is increment. Flow continues to the buffer copy done decision        468, and the loop continues until each line of the buffer 190 is        copied. When the buffer has been fully copied, flow terminates        at a buffer copy exit 476 point.        FIGS. 5A to 5C—Run-length Encoding Formats of the RHN Method

FIGS. 5A to 5C show use of a different structure than the presentinvention. FIGS. 5A to 5C show the formats for the run-length encodingof RHN. In FIG. 5A, a code byte 500, with its high order bit designatedas a flag bit 510.

FIG. 5B shows a repeat code 520 comprising a Boolean value one in itsflag bit 510 and a 7 bit count 530 in the remaining 7 low order bits.The seven bit count 530 can represent 128 values with a zerorepresenting “128” and 1 through 127 being their own value.

FIG. 5C shows a data code 550 comprising:

-   -   a Boolean value zero in its flag bit 510    -   two unused data bits: data bit 6 reference by 565 and data bit 5        reference by 570, and    -   five bits, data bits 4 to 0, reference by 575, 580, 585, 590,        and 595, respectively.

FIG. 5C shows that in every byte of the RHN data code 550 two bits areunused and one bit is used for the flag bit, so that only five of theeight bits are used for data. The remaining three bits are wasted bits560. The present invention uses a different structure by placing therepeat count in bits that the RHN format would not have used for data(U). The corresponding ZLN format, ZL5 (where N is 5, U is 3, and W is8), always uses five bits for data and the remaining 3 bits for therepeat count. In practice, repeat counts are small and often can fit in3 bits, so this embodiment of the present invention will result insuperior compression performance over the RHN method.

In addition, the present invention provides for a larger count when thebit filtering is larger. For example, the alternate ZLN format whereeach byte contains 4 data bits, ZL4 (where N is 4 and U is 4), allowsfor a four bits of repeat count. For example, in practice, ZL4 issuperior to RHN on a typical ultrasound image containing 16 shades ofgray.

FIGS. 5D to 5G—Doppler Improvement of ZLN Format

The ZLN format for encoding video signals (see for example FIG. 3A) hasone disadvantage. When the video stream being compressed changes fromonly grayscale to grayscale overlaid with Doppler enhancement, thecolors of the Doppler red and blue scales are also converted tograyscale. A Doppler improvement aspect of the present inventionprovides a variable means of encoding the Doppler enhanced image with inthe data bits of a ZLN format.

FIG. 3A shows the ZLN format comprising N 300 and U 301 that make up adata element with a bit length of W 302.

The Doppler improvement aspect encodes the Doppler values in the N 300portion of the ZLN format. However the scope of this invention shouldnot be limited to using this technique with ZLN formats only as othercompression formats are anticipated by this invention.

Doppler Encoding Format

The Doppler enhanced grayscale image must be captured as color data (seeFIG. 2B). If the red, green, and blue values are the same, then thepixel is gray (e.g. FIG. 2G). If the red, green, and blue values are notequal, then the pixel is representing a red scale or blue scale value(e.g. FIG. 2F). By comparing the values of the red, green, and bluevalues, a red scale or blue scale value is determined.

In FIG. 5D, two bits, S 503, are used to indicate which scale the pixelis from. The high bit order bit, S1501, is used to indicate that thevalue is a gray scale value: zero means grayscale (e.g. 515 in FIGS. 5Eand 541 in FIG. 5F), one means Doppler (e.g. 525 and 535 in FIG. 5E).The low order bit, S0 502, is used to indicate red scale or blue scale:zero means red scale (e.g. 525 in FIG. 5E), one means blue scale (e.g.535 in FIG. 5E). In the grayscale code 515, the value of the S0 bit is adon't care 512 and it can be coded as either a zero or a one. Incombination with the ZLN format, the remaining bits of W are used for arepeat count, C (FIG. 5G).

An alternate embodiment of this invention uses lookup tables ratherselecting the most significant bits. Instead of one encode (e.g. FIG.11A) and one decode table (e.g. FIG. 11B) as used in ZLN, a set oftables is used for each scale, gray, red, and blue, respectively. FIG.11C shows an example of a red scale decode table.

In an alternate embodiment, S0 is used as an additional bit of grayscaleresolution since S0 502 is not used in the grayscale case 545 (FIG. 5F).

In a method where W is the number of bits in a native machine data type,and N is the number of significant grayscale bits, two bits, S 503, areused to indicate which scale the pixel is from. The high bit order bitS1501 is used to indicate that the value is a gray scale value: zeromeans grayscale 515, one means Doppler 525 and 535. The low order bit,S0 502, is used to indicate red scale or blue scale: zero means redscale 525, one means blue scale 535. In the ZLN combination, theremaining unused bits, U, are used for a repeat count, C, such that Wequals 2+N+U (FIG. 5G).

N bits of the blue component of the pixel value is used to index into ablue encode table to obtain the encoded value, E. In the ZLN method, ifE is repeated, a repeat count, C, is incremented.

X 557 is a concatenation of S1, S0, E, and C.

In this embodiment, like the ZLN method, the pixels of the frame areprocessed pixel by pixel as disclosed in reference to FIGS. 4A and 4B.When a value, E, is not repeated X is placed in the next location in thecompression array with a repeat count, C, equal to one.

Three decode tables that correspond to the grayscale, red scale, andblue scale encode tables contain the data necessary to reconstruct theoriginal value for the appropriate image. If the target color imageformat is W*4 bit color (FIG. 2A), then the decode table has W bits foralpha 208, red 206, green 204, and blue 202 each, respectively. If thetarget color image format is W*3 bit color, then the alpha value is notused. If the image is W bit grayscale than only the grayscale value isused to create the decompressed and decoded image.

To decode and decompress, the encoded data is processed W bits at a timeas X. S1, S0, E, and C are extracted from X with appropriate masks andshifts. If S1 is zero indicating grayscale, E is used as an index intothe gray scale decode table. If S1 is one indicating Doppler and S0 iszero indicating red scale Doppler, E is used as an index into the redscale decode table (FIG. 11C). If S1 is one indicating Doppler and S0 isone indicating blue scale Doppler, E is used as an index into the bluescale decode table (not shown, but with the proper values in the bluebit column 1126). The decoded value is placed into the decoded anddecompressed image. Each X is processed in order until the compressedarray length, L, has been processed.

FIG. 6—Encoded Data Stream

FIG. 6 shows a series of exemplary decimal values 610 comprising a firstvalue 620 equal to decimal 0, a second value 622 equal to 0, a thirdvalue 624 equal to 0, a fourth value 626 equal to 0, a fifth value 628equal to 0, a sixth value 630 equal to 2, and a seventh value 632 equalto 10. The value of zero is merely exemplary and could be any binaryvalue. After the encoding step 130 (FIG. 1), the corresponding encodeddata 140 (FIG. 1) would be compressed down to three bytes of binary code640 comprising a first byte 650, a second byte 653, and a third byte 656each containing a merged value and count, (651, 652), (654, 655), and(657, 658), respectively. The first data 651 has a binary value of“00000” which equals the exemplary repeated decimal value. The firstcount 652 has a binary value “101” which equals decimal fiverepresenting the run-length of the repeating value in the first five ofthe decimal values 610. The second data 654 has a binary value of“00010” which equals the non-repeated decimal value two. The secondcount 655 has a value of 1. The third data 657 has a binary value of“01010” which equals the non-repeated decimal value ten. The third count658 has a value of 1.

FIG. 7—RHN Codes and Encoded Stream

FIG. 7 shows the same series of decimal values 610 (FIG. 6) comprisingthe first value 620 equal to decimal 0, the second value 622 equal to 0,the third value 624 equal to 0, the fourth value 626 equal to 0, thefifth value 628 equal to 0, the sixth value 630 equal to 2, and theseventh value 632 equal to 10. After encoding by RHN, the correspondingencoded data 140 (FIG. 1) would be compressed down to four bytes of RHNbinary code 740.

The embodiment of the present invention shown in FIG. 6 only requiresthree bytes to encode the same data. In this example, the presentinvention is 25% better than the RHN format.

FIGS. 8A and 8B—ZLN Formats

The ZLN aspect of the present invention provides for variable formats.The values of N 300, U 301, and W 302 can be dynamically changed betweenframes. For ease of communication a format is named with the prefix “ZL”and a digit representing the value of N. For example, “ZL5” refers to aformat where bit width of N is equal to 5. There are multiple values ofU depending of the W. To also specify the bit width of U a hyphen and anumber can be appended. For example, “ZL5-13” represents a format whereN=5 and U=13. “ZL5-3” is a common format and may be imprecisely referredto as “ZL5.”

FIG. 8A shows a number of formats with adjacent labels: ZL3 803, ZL4804, ZL5 805, ZL8 808, ZL9 809, and ZL12 812. Data bits are representedby “D,” and count bits are represented by “C”.

FIG. 8B shows how the most significant 3 bits of each color component(216, 214, and 212 of FIG. 2B) are extracted and formatted in ZL9-7Cformat (the “C” append indicates that the color is preserved). Withthree red bits represented by “R”, three green bits represented “G” andthree blue bits represented by “B”.

Decoding

To decode the compressed array, the decoder has a decode table thatcorresponds with the encode table. For W*4 bit color pixels, the decodetable contains the appropriate alpha, red, green, and blue values. ForW*3 bit color pixels, the alpha value is not used. The compressed arrayis processed W bits at a time as X. The repeat count, C, is extractedfrom X by masking off the data value (C=X & (((2**N)−1)<<U)). Theencoded value, E, is extracted from X by masking off the count (E=X &((2**U)−1)). The encoded value, E may be used to index into thedecryption. The decoded pixels are placed in a reconstructed image andrepeated C times. Each element of the compressed array, A, is processeduntil its entire length, L, has been processed.

FIG. 9—Decode Flowchart

FIG. 9 illustrates the decode flowchart 920 which presents the detailsof the decryption embodiment of the decode step 160 (FIG. 1) and theimage reconstitution step 180 (FIG. 1).

The decoding begins at a decode entry 900. In a “decode initialization”step 901, a repeat counter C is set to one, an encoded length L is setto the value obtained with the encoded data 140 (FIG. 1), and an index Iis set to 0. Next, a “get code” step 902 obtains a signed byte X fromthe encoded data 140 (FIG. 1) array A. The index I is incremented. Thecount (for example the 3-bit count 380 as shown in FIG. 3B) is extractedfrom X by masking off the data bits and placed in the repeat counter C(C=X & ((2**N)−1<<U). The value of E is extracted from X by masking offthe count bits (E=X & (2**U)−1). In practice, the count mask and valuemask can be pre-computed with the following two lines of code in the Cprogramming language:

valueMask = −1 << U; countMask = ~valueMask;

In this illustrative decryption embodiment of the present invention,flow goes to a “decode lookup” step 908 where the value of E is used toindex into the decode table 1110 (FIG. 1I, shown as decryption table) toobtain a pixel value V. In the other embodiments where E is notencrypted, E is used as V and step 908 is bypassed. Flow continues to a“check zero count” 909 decision.

The 909 decision always fails the first time ensuring that a place pixelstep 910 is executed. The place pixel step 910 places the pixel value Vin the next location of the decompressed image and decrements the repeatcounter C and returns to the 909 decision. The pixel value V is placedrepeatedly until C decrements to zero. Then the 909 decision branchesflow to a “reset counter” step 914. At step 914 the repeat counter isreset to 1.

Flow continues to the “check length” 916 decision where the index I iscompared to the encoded length L to determine if there are more codes tobe processed. If I is less than L flow returns to step 902, otherwisethe decode process terminates at a “decode exit” 918.

The entire decode process is repeated for each encoded frame image.

FIG. 10—Interpolation

FIG. 10 illustrates interpolation when two adjacent pixels 1010 and 1012and two subsequent row adjacent pixels 1014 and 1016 are stretched toinsert a new row and column of pixels.

Pixels 1052, 1054, 1056, 1058 and 1060 are inserted due to theenlargement of the image. Their values are calculated by averaging thevalues of the two pixels above and below or to the left or the right ofthe new pixel. A preferred sequence is calculation of:

-   1. 1052 between 1010 and 1012-   2. 1054 between 1010 and 1014-   3. 1058 between 1012 and 1016-   4. 1056 between 1054 and 1058    Pixel 1060 can be calculated on the interpolation for the subsequent    row.    FIG. 11—Encryption

By using corresponding encoding and decoding tables the data can beencrypted and decrypted without using actual values. Encryption providesa level of security for the encoded data 140 while in storage ortransit.

FIG. 11 shows an example of an encryption table 1100, where N is 3 and Wis 8, and a decryption table 1110, where N is 3 and U is 5.

The encode table 1100 is 2 to the power of N in length. If the targetcolor image format is W*4 bit color, then the decode table 1110 has Wbits for alpha, red, green, and blue each, respectively. If the targetcolor image format is W*3 bit color, then the alpha value is not used.If the image is W bit grayscale then only the grayscale value is used tocreate the decompressed and decoded image.

The corresponding table elements are mapped to each other. For example,0 could encode to 22 as long as the 22^(nd) element of the decode tablereturns (θxff<<24|θ<<16|θ<<8|θ).

When these versions of the tables are used, the encode and decodeprocesses and their speed of execution are substantially the same butthe encoded data 140 (FIG. 1) becomes a cipher and has a higher level ofsecurity. It should be recognized by one with ordinarily skill in theart that there are other embodiments of the present invention withdifferent encryption/decryption table rearrangements.

FIGS. 12A through 12D—Compression and Decompression Devices

FIGS. 12A and 12B show devices for compressing and decompressing,respectively, a stream of video frames.

FIG. 12A shows a video signal 1215 being compressed and encoded by acompressor 1210 to form an encoded data stream 1235, which is sent to anI/O device 1240. The video signal 1215 comprises a series of videoframes 1200, shown as first video frame 1205 a, second video frame 1205b, . . . through nth video frame 1205 n. The encoded data stream 1235comprises a series of encoded data 1220, shown as first encoded data1225 a, second encoded data 1225 b, . . . , through nth encoded data1225 n.

FIG. 12B shows an input encoded data stream 1245 being received from anI/O device 1240, and then, decoded and decompressed by a decompressor1250 to form a video sequence 1270. The input encoded data stream 1245comprises received encoded data 1238, shown as first received encodeddata 1230 a, second received encoded data 1230 b, . . . , through nthreceived encoded data 1230 n. The video sequence 1270 comprises a seriesof decoded video frames 1268, shown as first decoded video frame 1260 a,second decoded video frame 1260 b, . . . , through nth decoded videoframe 1260 n.

FIG. 12C shows an embodiment where the I/O device 1240 of FIGS. 12A and12B is a storage medium 1280. The encoded data stream 1235 from thecompressor 1210 is stored in the storage medium 1280. The storage medium1280 provides the input encoded data stream 1245 as input to thedecompressor 1250.

FIG. 12D shows an embodiment where the I/O device 1240 of FIGS. 12A and12B is a communications channel 1290. The encoded data stream 1235 fromthe compressor 1210 is transmitted over the communications channel 1290.The communications channel 1290 provides the input encoded data stream1245 as input to the decompressor 1250.

FIGS. 13A through 13B

FIG. 13A shows details of an embodiment of the compressor 1210, whichcomprises a video digitizer 1310, a video memory 1330, an encodingcircuit 1350, and encoded data 1370. Each video frame 1205 in the seriesof video frames 1200 is digitized by the video digitizer 1310 and storedalong path 1320 in the video memory 1330. The encoding circuit 1350access the digitized video frame via path 1340 and outputs the encodeddata 1370 along path 1360. The encoded data 1225 corresponding to eachvideo frame 1205 is then output from the compressor 1210.

FIG. 13B shows further details of an embodiment of the encoding circuit1350. A pixel sub-sampler 1380 scans each pixel from the digitized videoframe in the video memory 1330. A pixel index 1332 is used to drive ascan 1331 signal to select each pixel from the video memory, in apredetermined sequence. A novel aspect of the present invention is thatthe compression method can be accomplished with a single scan of thevideo memory for each frame. The pixel sub-sampler 1380 selects apredetermined number of bits from each pixel and outputs the data valuealong path 1385. Alternatively, the pixel sub-sampler 1380 encodes thesub-sampled data by using a lookup table similar to FIG. 11A. Differentpixel sub-samplers 1380 will be discussed in reference to FIGS. 14through 13J. The data/count 1390 unit increments the count each time theoutput of the pixel sub-sampler 1380 is the same; otherwise, when theoutput of the pixel sub-sampler 1380 is different (or when the counterreaches the maximum count value, the data and count are combined as acode and output along path 1395 to the encoded data 1225 for the framecurrently in the video memory 1330. The location of the code in theencoded data 1225 is selected by the code index 1392 signal.

FIG. 14 Pixel Sub-Sampling

FIG. 14 shows further details of a generic pixel sub-sampler 1380. Whena pixel is scanned from video memory along path 1340, it has an originalpixel bit width, P. A pixel extractor 1382 extracts a subset of bitsfrom each pixel with a value bit width, V, along value path 1383. Thevalue bit width V is less than the pixel bit width P. A coder 1384 takesthe V bits from the pixel path 1383 and outputs a code with an encodedbit width, E, as the data value along path 1385. One embodiment of thecoder is a null coder, or pass-through coder. Another embodiment of thecoder uses an encryption table to encrypt the data value as an encrypteddata value.

FIG. 15—Decompressor Elements

FIG. 15 shows details of an embodiment of the decompressor 1250, whichcomprises a decoding circuit 1510 which inputs received encoded data1230 and outputs decoded pixel values 1520 to an image memory 1540. Adecoder pixel index 1530 selects the location in the image memory 1540to store the decoded pixels values 1520. The image memory 1540 deliverseach decoded video frame 1260 to the video display.

FIGS. 16A and 16B—Parameters Altered by a Remote Receiver

FIG. 16A shows a system for setting width, height, frame rate,brightness, and contrast in a transmitter 1600 which are variablyaltered by a receiver 1610. The receiver sends commands to thetransmitter 1600 via setting control path 1615. The commands alter thetransmitter settings 1660.

The settings 1660 include brightness 1661, contrast 1662, height 1663,width 1664, and frame rate 1665. The brightness 1661, contrast 1662,height 1663, and width 1664 setting alter the attributes of each frameas it is digitized in a frame sub-sampler 1620. The brightness 1661 andcontrast 1662 settings alter the video digitizer 1310 (FIG. 13A) as itsenses the video frame. The height 1663 and 1664 allow for optionallyselecting a subset area of each frame; this is area sub-sampling.Alternatively, height 1663 and 1664 allow for optionally selecting asubset of pixels from an array of pixels that make up a single frame, byskipping pixels in a row or by skipping rows; this is imagesub-sampling. The frame rate 1665 setting alters the frame selector 1670which drives the frame select indicator 1675 to optionally sub-sampleframes from a sequence of video frames; this is frame sub-sampling.

The frame sub-sampler 1620 outputs a selected frame 1630 along path1621. The transmitter pixel sub-sampler 1640 scans the selected frame1630 getting each pixel from frame 1632 and outputs data values alongpath 1642 to a run length encoder 1650. The encoded data stream 1235 isthen transmitted to the remote receiver 1610.

FIG. 16B shows additional elements of a system for setting the number ofpixel bits in an alternate transmitter 1690 which is variably altered bya receiver 1610. The receiver sends commands to the transmitter 1600 viasetting control path 1615. The commands alter the transmitter settings1660. The settings include a number of pixel bits setting 1680 whichaffect the number of bits selected by the transmitter pixel sub-sampler1640. The pixel sub-sampler 1640 could be any pixel sub-sampler, forexample, see FIG. 14 through 13J and 14A through 14C. The transmitterpixel sub-sampler 1640 scans the selected frame 1630 (as in FIG. 16A)getting each pixel from frame 1632 and outputs data values along path1642 to a run length encoder 1650. The encoded data stream 1235 is thentransmitted to the remote receiver 1610.

These embodiments illustrate the novel feature of the present inventionof allowing a user at a remote receiver 1610 to control aspects of thetransmitter 1600 or 1690 from a remote location, including brightness,contrast, frame dimensions, frame rate, image area, and the type ofcompression used.

FIG. 17—Further Lossless Compression Step

FIG. 17 shows a lossless compression step for further compressing anencoded data buffer. After a run-length encoding step 1700 in thetransmitter, a run-length encoded output 1710 can be further processedwith a further lossless compression step 1720 resulting in furtherlossless compression output 1730. The further lossless compression step1720 could be implemented as a variable length coding, arithmeticcoding, or other compression step known in the art.

FIG. 18—Image Stretching

FIG. 18 shows images being enlarged by stretching. An unstretched frame1800 is stretched during stretching step 1820 resulting in an enlargedimage 1810. When a frame is image sub-sampled or area sub-sampled, theremaining data can be stretched to fill the full display area on thereceiver 1610. This results in an interpolated image or magnified image,respectively.

Distinguishable Characteristics

Most video images contain regions that are distinguishable from theother pixels that make up an image. Sometimes the distinguishingcharacteristic is the importance of the region to the viewer. In videoconferencing, for example, the face region may be of most importance tothe viewer. In medical imaging such as ultrasound, the generated imagein the center of the display may be of most importance to the viewer.Sometimes the distinguishing characteristic is the compressibility ofthe regions. Sometimes the distinguishing characteristic is the colordepth of the regions. Sometimes the distinguishing characteristic is therate of change of the regions. Other distinguishing characteristics,such as markers, have been mentioned above.

The following are additional examples of distinguishablecharacteristics.

When watching a sporting event the motion of the players or the ball ismuch more important than the playing surface or the background audience.If the region of interest is generally distinguishable based on colorrange, brightness range, or position on the screen, those regions couldbe updated more quickly or compressed with more detail while theremaining regions of less importance are compressed or displayed in away that requires less resources.

When watching a news broadcast or interview the region of the “talkinghead” or a graphic display may be the region of most interest toviewers.

A region of solid color or grayscale value compresses more efficientlythan a series of varying values. This is true of the ZLN compressionmethod. If the regions are distinguished based on their compressibility,different compression methods can be applied to each region.

Grayscale pixel values can be stored in 8 bits while the correspondingquality of color pixel is often stored in 24 or 32 bits. If the regionsare distinguished based on their storage requirements (also known ascolor depth, or bit depth), a significant space or bandwidth saving canbe made.

A Doppler enhanced image such as a weather map or an ultrasound medicalimage is synthesized by the Doppler circuitry. In the case of a weathermap, the underlying image does not change but the Doppler enhancedvelocity scales do change from frame to frame. In the case of Dopplerenhanced ultrasound image, the underlying grayscale ultrasound imagechanges more frequently than the circuitry can calculate and display theDoppler information. If the Doppler and non-Doppler regions areprocessed separately the overall effective compression sizes andtransmission times can be reduced.

Plane Separation

An aspect of the present invention teaches that by separating eachdistinguishable region of a video frame into separate planes andapplying different compression methods that are optimal for eachresulting plane, the overall effective compression and transmissionspeeds can be increased.

FIG. 19A

FIG. 19A shows an example of an original image 1900. In this simpleexample, the image is comprised of a grayscale region 1902 that containsa grayscale triangle 1904, a red oval 1910 and a blue circle 1912. Thered oval 1910 and the blue circle 1912 are distinguishable from the restof the image because their red, green, and blue component values aredifferent and need to be stored in 24 (or 32) bits. The remaining regionhas equivalent values in the red, green, and blue component values foreach pixel and therefore can be reduced to 8 bits and still maintain thefull quality. In this example, the grayscale pixels are separated alonga first path 1915 to a first plane 1920, and the color pixels from thered oval 1910 and the blue circle 1912 are separated along a second path1925 to a second plane 1930. In the first plane 1920, the unselected(e.g. color) pixel locations are filled with a mask value, such asblack, leaving a sold region of repeating values that will compress wellwith a compression method such as ZLN. In the second plane 1930, theunselected pixels are filled with a mask value, such as white. The colorchosen for the mask value generally does not matter because those pixelsare overlaid (in the base plane) or ignored (in the overlay plane) priorto display.

The mask values could be any value that is known to be a mask by thedecompressor at decompression time. The mask value can be predeterminedor encoded with the compressed plane.

As stated above, the distinguishing characteristic is not limited tograyscale versus color and could vary over time in the same videoprogram. It is also within the scope of this invention to separate theoriginal image 1900 into more than two planes.

FIG. 19B

FIG. 19B illustrates the compression, storage or transmission,decompression, and composition of the separated planes. The originalimage 1900 is separated into two or more planes as explained above. Eachplane is then compressed with a compression method that is optimum forits distinguishing characteristics. These compression methods could be:one well known in the art, one of a co-pending patent application, or acurrent or future proprietary one. The first plane 1920 is compressedusing a first compression method 1940 resulting in a first encoded data1950. The second plane 1930 is compressed using a second compressionmethod 1945 resulting in a second encoded data 1955. The encoded datacould be stored in a storage medium such as a computer memory, a harddrive or CD-ROM, or transferred across a network as a part of a videostream. A composite buffer 1960 represents either the encoded data on astorage medium or the data being transferred over the network, and mayalso include audio or textual data associated with the video.

Upon decompression, the first encoded data 1950 is decompressed with afirst decompression method 1970 that corresponds with the firstcompression method 1940, resulting in a first decoded image 1980. Thesecond encoded data 1955 is decompressed with a second decompressionmethod 1975 that corresponds with the second compression method 1945,resulting in a second decoded image 1985. Typically, the first decodedimage 1980 is the base image and will be composed along a copy path 1990with the second decoded image 1985 along an overlay path 1995 whichremoves the masked pixels and overlays the base image with the remainingpixels to form a combined image 1999. Except for effects of the variouscompression methods, the combined image 1999 will be substantiallyrepresentative of the original image 1900.

FIG. 20

FIG. 20 shows a flow chart for an embodiment of the method of thepresent invention. The chart begins at an entry point 2000 and continuesalong path 2002 to a “get pixel” step 2004 where the next pixel from theoriginal image 1900 is obtained. Flow continues along path 2006 to a“which plane” decision 2008. If the distinguishing characteristic of thecurrent pixel indicates that it should be separated to the first plane,flow continues along path 2010 to an “add to first buffer” step 2012where the pixel is added to the first buffer. Flow continues, along path2014, to a “put mask in second buffer” step 2016 where the mask valuefor the second plane is written to the second buffer. Flow continuesalong path 2018 to an “increment buffer pointers” step 2030.

However, if the distinguishing characteristic of the current pixelindicates that it should be separated to the second plane 1930, flowcontinues along path 2020 to an “add to second buffer” step 2022 wherethe pixel is added to the second buffer. Flow continues, along path2024, to a “put mask in first buffer” step 2026 where the mask value forthe first plane is written to the first buffer. Flow continues alongpath 2028 to the “increment buffer pointers” step 2030.

After separating the pixel to the appropriate plane and masking theother plane, at the “increment buffer pointers” step 2030, the pointersto the input buffer, the first buffer, and the second buffer areincremented to point to the next location in each respective buffer.Flow continues along path 2032 to a “done” decision 2040.

If there are more pixels in the input buffer, flow continues, along path2050, to the “get pixel” step 2004 until each pixel has been separated.After all the pixels are separated into their respective planes, flowcontinues along path 2060 to a “compress first buffer” step 2062 andthen along path 2064 to a “compress second buffer” step and thenterminates along path 2068 at an exit point (“finish”) 2070.

It would be clear to one skilled in the art that the order of abovesteps could be changed with the same result. For example, anywhere anaction is done with the first buffer and then another action to thesecond buffer, the steps could be done in the reverse order.

FIG. 21

FIG. 21 illustrates an example of how different planes can bedecompressed at different frame rates. In this example, four originalimages: the original image 1900, a second original image 2101, a thirdoriginal image 2111, and a fourth original image 2121, respectively,make up a sequence of video frames, an input video stream 2180.

As explained above, the original image 1900 (labeled I1) is separatedinto the first plane 1920 (labeled I1-1) and the second plane 1930(labeled I1-2). The first plane 1920 is compressed, transferred tostorage or across a network, and decompressed as the first decoded image1980 (labeled 01-1). The combined steps of compression, transfer, anddecompression are represented by a first transfer 2100.

The second plane 1930 is compressed, transferred to storage or across anetwork, and decompressed as the second decoded image 1985 (labeled01-2). The combined steps of compression, transfer, and decompressionare represented by a second transfer 2110.

The first and second decoded images, 1980 and 1985 respectively, arecombined as explained above to form the combined image 1999 (labeled01).

The second original image 2101 (labeled I2) is separated into a secondfirst plane 2102 (labeled I2-1) and a second second plane 2103 (labeledI2-2). The second first plane 2102 is compressed, transferred to storageor across a network, and decompressed as a second first decoded image2105 (labeled 02-1). The combined steps of compression, transfer, anddecompression are represented by a third transfer 2120.

The second second plane 2103 is not transferred. In the case where theregion of the second plane is of less interest to the viewer, this planeof this frame can be skipped (or dropped) to reduce the storage space ornetwork bandwidth required. In the case of Doppler enhanced ultrasound,this plane may have not been changed by the generating circuitry (asexplained above) and when the lack of change is detected (by comparingwith the data of the previous frame, or by notification from theultrasound device) this plane of this frame can be skipped. In thegeneral case where encoded color data is more resource intensive thanencoded grayscale data, the encoded color data may simply be skippedperiodically just to reduce the overall resources required for the videostream.

Because the second second plane 2103 is not transferred, the secondfirst decoded image 2105 (which is newer than the first decoded image1980) is combined with the older second plane data, in this case thesecond decoded image 1985 as shown by a “reuse of second decoded image”path 2106. This results in a second combined image 2107 (labeled 02).

The second combined image 2107 is not necessarily a close representationof the second original image 2101 but is a close approximation and willquickly be replaced by a more accurate combined image 2117.

A third original image 2111 (labeled 13) is handled in the same manneras the original image 1900. The third original image 2111 is separatedinto a third first plane 2112 (labeled 13-1) and a third second plane2113 (labeled 13-2). The third first plane 2112 is compressed,transferred to storage or across a network, and decompressed as a thirdfirst decoded image 2115 (labeled 03-1). The combined steps ofcompression, transfer, and decompression are represented by a fourthtransfer 2130.

The third second plane 2113 is compressed, transferred to storage oracross a network, and decompressed as a third second decoded image 2116(labeled 03-2). The combined steps of compression, transfer, anddecompression are represented by a fifth transfer 2140.

The third set of first and second decoded images, 2115 and 2116respectively, are combined, as explained above, to form a third combinedimage 2117 (labeled 03).

A fourth original image 2121 (labeled I4) is handled in the same manneras the second original image 2101. The fourth original image 2121 isseparated into a fourth first plane 2122 (labeled I4-1) and a fourthsecond plane 2123 (labeled I4-2). The fourth first plane 2122 iscompressed, transferred to storage or across a network, and decompressedas a fourth first decoded image 2125 (labeled 04-1). The combined stepsof compression, transfer, and decompression are represented by a sixthtransfer 2150.

The fourth second plane 2123 is not transferred, as explained above,regarding the second second plane 2103.

Because the fourth second plane 2123 is not transferred, the fourthfirst decoded image 2125 (which is newer than the third decoded image2115), is combined with the older second plane data, in this case thethird second decoded image 2116, as shown by a “reuse of third decodedimage” path 2126. This results in a fourth combined image 2127 (labeled04).

The fourth combined image 2127 is not necessarily a close representationof the fourth original image 2121, as explained above.

Thus, in this example, an input video stream 2180 (comprising frames I1,I2, I3, and I4) is compressed, transferred, and decompressed to anoutput video stream 2190 (comprising frames 01, 02, 03, and 04).However, only six of the eight separated planes are transferred. In somecases, especially when the characteristics of the second plane areresource intensive, this will result in a substantial reduction in theamount of resources required to store or transmit the video stream,without substantially reducing the value or quality of the video stream.

Although not shown in FIG. 21, the present invention teaches that imagesmay be sampled at two independent rates where the first plane isextracted only from images at a first sample rate and the second planeis extracted only from images at a second sample rate. For example, ifan ultrasound machine is known to generate the grayscale image at 24frames per second and to generate the Doppler overlay at 10 frames persecond, the grayscale plane can be extracted from the current image 24times per second, and the color plane can be extracted 10 times asecond. It is anticipated that the image generation circuitry of theultrasound device could signal the image sampler each time therespective images are generated. It is also anticipated that thegrayscale base image and the Doppler overlay image could be fed directlyinto the present invention along the first path 1915 and the second path1925 skipping the step normally performed by the ultrasound device thatcreates the original image 1900.

FIG. 22A

FIG. 22A illustrates a system for user selection of separation regions.In this alternate embodiment of the present invention, the region of theimage to be separated is selected by a user input 2200. The user couldbe either a video producer who is compressing a video stream forstorage, a viewer who is receiving a video stream at a remote site, abroadcaster who is transmitting a video stream, or similar person who isempowered with the ability to control the plane separation. A system ofallowing user input of the separate regions comprises the following:

-   -   1) a display 2230 that allows the user to see the video and        optionally view a selection indicator. The display could show        both the uncompressed video and the results of the separation        and separate compression methods and frame rates.    -   2) an image source 2220 that provides a stream of multiple        original image 100 frames.    -   3) the user input 2200 that allows the user to select one or        more regions that are to be separated. The user selection        becomes the distinguishing characteristic used in applying the        present invention.    -   4) a display generator 2210 that combines the image source 2220        with the selection indicator from the user input 2200 to drive        the display 2230. The display generator could also display the        results of the separate plane compression method of the present        invention.    -   5) a control data 2240 is generated by the display generator        2210 based on the user input 2200. The control data 2240        describes the user's selection. Typically, the control data        would be encoded into a protocol understood by the means for        separating the planes. For example, if the selection region was        a rectangle, the coordinates for opposing corners of the        rectangle could be encoded along with the indication that the        selection shape was a rectangle.

The user input 2200 is connected to the display generator 2210 via auser input path 2205. Examples of user input 2200 devices includekeyboards, mice, trackballs, touchpads, touch screens, eye motionsensors, voice commands, and the like. The user input path 2205 for eachof these types of devices are known in the art.

The image source 2220 is connected to the display generator 2210 via animage source path 2215. The image source path 2215 is known in the art,including, but not limited to, composite video, S-Video, DV, mini-DV,video digitizers, USB, FireWire, serial port, parallel port, and thelike.

The display 2230 is connected to the display generator 2210 via adisplay path 2225. Typically the display generator would be a computingdevice such as a desktop computer or a TV set top box and would beconnected to the computer monitor or television set, respectively. Thepresent invention anticipates that there are many embodiments of thedisplay generator and the display including, but not limited to, videophones, satellite TV, cable TV, video conferencing systems, ultrasoundmachines, weather displays, air traffic control systems, law enforcementsystems, military systems, game consoles, and the like.

The control data 2240 is generated by the display generator 2210, alonga control data path 2235. If the user is using a single device tocompress the video stream, the control data 2240 is passed internally tothe means for separating the planes. If the user is a remote viewer, asin the video conferencing example, the control data 2240 is sent via thenetwork, or the video conferencing connection, to the transmitter wherethe means for separating the planes is located, allowing the planes tobe separated and then compressed and transferred separately.

The user input 2200 allows the user to specify the shape, size, orlocation of the selection region. The user could manipulate the userinput 2200 to enable and disable the display of a selection indicator.The selection indicator could be the outline of the region iscontrasting color. The contrasting color could be a solid color such asred, or it could be filter applied to each pixel of the selectionindicator. Examples of filters are inverse (where the inverse colorvalue replaces the pixel), lighten (where each pixel value is madebrighter), darken (where each pixel value is made darker), or a similarfilter as known in the art.

One user input 2200 example of particular interest is the eye motionsensor, because as the video stream is being viewed the eye movement ofthe user can be immediately detected and used to select the region ofinterest of the viewer.

FIG. 22B through FIG. 22G

FIG. 22B through 22G show examples of various selection shapes. FIG. 22Bshows a selection shape that matches a human head and shoulders, namelya bust region 2250. The bust region 2250 is separated into one planewhile a background region 2260 is separated into another plane.

FIG. 22C shows an oval region 2270. The oval region 2270 is separatedinto one plane while a background region 2260 is separated into anotherplane.

FIG. 22D shows a rectangular region 2290. The rectangular region 2290 isseparated into one plane while a background region 2260 is separatedinto another plane.

FIG. 22E shows a combined region 2299, composed of the union of the ovalregion 2270 and the rectangular region 2290. The combined region 2299 isseparated into one plane while a background region 2260 is separatedinto another plane.

As explained above, the user could manipulate the user input 2200 toselect from various predetermined shapes and modify the shape, size, andlocation of the selected region. The user can also draw a region of anyshape. More that one region could be selected. Some embodiments of thepresent invention will employ multiple selections to separate out threeor more planes. All of these selections and how they are displayed onthe display 2230 can be controlled by the user.

The present invention anticipates that if the area of the selectionregion is smaller than the entire frame, only the area of the selectionregion needs to be processed. For example, if the selection region isthe rectangular region 2290 (shown if FIG. 22D), only the subsetrectangle without the surrounding pixels can be compressed and encodedwith the relative coordinates of the selection region. The surroundingpixels can be assumed by the decompressor to be mask values.

Automatic Switching between Grayscale Only and Doppler Enhanced Formats

As disclosed, for example in co-pending U.S. application Ser. No.09/321,922 regarding its FIG. 2, the video image capture device receivesa stream of video images (1200) from a video source. The compressor 1210is configured to compress the stream of video images 1200 therebycreating a compressed stream of video images (e.g. 1235). This is alsoshown in FIG. 12C. Video parameters such as the compression algorithmare included within the video settings. In one embodiment of thisinvention, the compressor, which is determining the scale of each pixel,can count the number of Doppler enhanced pixels in each frame, or image,of the video stream. As discussed above, whether a pixel is grayscale orDoppler enhanced is determined by comparing the red, green, and bluecomponent values. If no Doppler pixel is found in an image, a standardgrayscale format, such as ZLN format, can be used as the compressionmethod of the compressor (i.e. Doppler enhanced encoding can beautomatically switched off). When a Doppler enhanced pixel is found in asubsequent frame of the video stream by the same counting mechanism, thecompressor can automatically be switched to use the Doppler enhancedformat described above. The compressor will continue to use thiscompression method until the counting mechanism fails to detect aDoppler enhanced pixel, then the grayscale only compression method canbe switched on again.

ADVANTAGES

Execution Speed

The methods of the present invention provide a decrease in theprocessing time required to process images that are being input oroutput. This decrease in processing time allows for video images to beenhanced, compressed, and encrypted in real time. The time saved bythese methods can be used to execute more efficient compressionalgorithms that may in turn reduce the bandwidth required to transferthe encoded data between computers or may reduce the space needed tostore the encoded data.

Reduced Memory Requirements

The selection of a subset image 194 from a super image 490 (FIG. 4C)reduces the amount of memory needed to hold the data being processed.

Noise Filtering and Image Enhancement

The removal of the least significant bits of pixel values results inhigh quality decompressed images when the original image is generated byan electronic sensing device, such as an ultrasound machine, which isgenerating only a certain number of bits of grayscale resolution. Byvariably altering the number of most significant bits, various filterscan be implemented to enhance the image quality. Such a noise filter canbe beneficial when the image is generated by an imaging technology suchas radar, ultrasound, x-ray, magnetic resonance, or similar technology.Variations can be made to enhance the perceived quality of thedecompressed image. Therefore, altering the number of data bits selectedand altering the width of the repeat count is anticipated by thisinvention and specific values in the examples should not be construed aslimiting the scope of this invention.

Dynamic Variable Formats

While a video stream is being viewed, a viewer on the decoding end ofthe transmission can vary the settings for the compressor. Differenttradeoffs between image spatial and temporal quality can be made. As thecontents of the video signal change an appropriate format can beselected. Control signals can be sent back to the compressor via acommunications link.

While a video stream containing Doppler enhancement is being viewed, aviewer on the decoding end of the transmission can vary the settings forthe compressor. Different tradeoffs can be made. For example, moreDoppler detail can be chosen with slower frame rate.

Automatic Switching

If no Doppler pixel is found in an image, a standard ZLN format can beused (i.e. Doppler enhanced encoding can be automatically switched off).When Doppler enhancement again appears in the video stream (asrecognized by the detection of a Doppler pixel in a frame), the Dopplerenhanced encoding can automatically be switched on again.

General Purpose

The lossless compression of the sampled data achieved by a preferredembodiment of the present invention results in high quality videostreams that have general purpose application in a number of areasincluding, without limitation, medical, aviation, weather traffic, videoconferencing, surveillance, manufacturing, rich media advertising, andother forms of video transmission, storage, and processing.

Lossless Nature/No Artifacts

Once the analog signal is sub-sampled and filtered to select a filteredpixel value that eliminates some of the real world defects, the methodsof the present invention compress and decompress the data with noirreversible data loss. Unlike JPEG and MPEG, the decompressed imagenever suffers from artificially induced blocking or smearing or otherartifacts that are result of the lossy compression algorithm itself. Asa result even a small sub-sample of the image remains clear and true tothe perceived quality of the original image.

Optimal Encoding

The present invention also provides a method for separating a videoimage into distinguishable regions. Each region can be encoded,compressed, and transferred in a manner that is optimal for itsdistinguishing characteristics.

Reduced Size

The present invention may also reduce the size of an encoded videostream by optimally encoding separated planes. The reduced size saves inthe usage and cost of storage devices and computing and networkingresources.

Reduced Bandwidth

The present invention may also reduce the bandwidth required to transfera compressed video stream. Both transfers within a computer system to astorage device, such as a hard disk, tape drive, and the like, andtransfers between a transmitter and a receiver over a network, such as aLAN, the Internet, a television network, and the like, are improved.This improvement comes from: improved compression, separate frame ratesfor separate planes, and selective frame dropping.

This improved bandwidth allows for the regions of interest to bedisplayed at a higher quality of resolution and motion while reducingthe requirements and cost of a high bandwidth connection or a connectionwith reduced traffic. For example, the present invention allows a videosteam that previously had to be sent over a 1.54 Mb T1 line to be sentover a much less costly and much more prevalent DSL, cable modem, or 56Kb modem connection.

Efficient Doppler Handling

The present invention also provides efficient methods for handlingDoppler enhanced images. This allows for lower cost storage of weather,air traffic, and medical images. It also allows for enhanced quality ofimages.

Automatic Detection of Eye Movement

The present invention also provides for automatic detection of the eyemovement of a user. This allows for a live television broadcast orwebcast to be transmitted using the methods of the present invention.

Automatic Detection of Regions of Interest

The present invention allows for automatic detection of regions ofinterested based on characteristics of the images, such as color depth,areas of change, external source information, or other distinguishingcharacteristics.

Automatic Marker Detection

The present invention also provides for detection of markers that can beautomatically detected to determine regions of greater interest or ofimproved compressibility. These allow for automatic application of themethods of the present invention. For example, in the case of abroadcast of a sporting event, markers can be used to determine areas ofinterest so that the important motion of the athletes is preserved whenthe rate of change normally would prevent a satellite from keeping upwith the data. This results in a much more pleasing experience forviewers across the nation.

CONCLUSION, RAMIFICATION AND SCOPE

Accordingly, the reader will see that methods the present inventionprovides a means of reducing the processing time and computer resourcesneeded to process images being input or output.

Furthermore, the present invention has additional advantages in that itprovides a means for reducing the space required in a storage medium.

Furthermore the compression and decompression steps of the presentinvention provide a means of digitally compressing a video signal inreal time, a means of digitally compressing Doppler enhanced videosignal in real time, communicating the encoded data stream over atransmission channel, and decoding each frame and displaying thedecompressed video frames in real time. The present invention alsoprovides a method of distinguishing between regions of an image,separating and masking the original image into multiple image planes,and compressing each separated image plane with a compression methodthat is optimal for each plane's characteristics. From a video stream,separate image streams can be stored or transmitted at different rates.

Furthermore, the present invention has additional advantages in that:

-   -   1. it provides a means of filtering real world defects from the        video image and enhancing the image quality;    -   2. it allows for execution of both the compression and        decompression steps using software running on commonly available        computers without special compression or decompression hardware;    -   3. it provides decompressed images that have high spatial        quality that are not distorted by artifacts of the compression        algorithms being used;    -   4. it provides a variably scalable means of video compression        and of adding Doppler enhancement; and    -   5. it provides a means for reducing the space required in a        storage medium.

While my above descriptions contain several specifics these should notbe construed as limitations on the scope of the invention, but rather asexamples of some of the preferred embodiments thereof. Many othervariations are possible. For example, the memory copy algorithm can beimplemented in a number of ways without limiting the scope of thisinvention to the use of a particular implementation. For example, bitordering can be altered and the same relative operation, relativeperformance, and relative perceived image quality will result. Also,these processes can each be implemented as a hardware apparatus thatwill improve the performance significantly. In another example, framedifferencing may be applied to the input stream to select a subset of aframe to be the original image, or a post processing step could be addedto remove artifacts introduced by a particular decompression method.

Accordingly, the scope of the invention should be determined not by theembodiments illustrated, but by the appended claims and their legalequivalents.

1. A method of increasing image processing performance, comprising:copying image data of a selected subset of an entire image existing inan I/O RAM into a copy of the selected subset in a buffer in a mainmemory by multiple calls to a memory copy function copying each imageline of the selected subset to a buffer line of the buffer; performingCPU intensive operations on the copy selected subset in the main memoryand not on the image data in said I/O RAM to generate a processed imageand storing the processed image in the buffer; and copying the processedimage from the buffer in the main memory to the I/O RAM by multiplecalls to the memory copy function copying each buffer line of the bufferto an image line to generate a copy of the processed image in the I/ORAM; wherein performing CPU intensive operations comprises bit-wisesub-sampling pixels of the copy selected subset to extract correspondingsubsets of pixel bits from the pixels; counting consecutive repeatedinstances of various pixel values represented by the extracted subsetsof pixel bits; and outputting a series of data elements for the pixelsto form the processed image, based on the extracted subsets of pixelbits; wherein a data element comprises a data field and a run-lengthfield, wherein the data field comprises a pixel value in a rangespanning from zero to a maximum value represent-able by a number ofpixel bits sub-sampled from a pixel, and wherein the run-length fieldcomprises a consecutive repeat count of the pixel value in the datafield.
 2. The method of claim 1, wherein the entire image comprises atelevision video image including a visible portion visible on a displaydevice and a non-visible portion not visible on the display device, andthe selected subset of the entire image substantially comprises only thevisible portion.
 3. The method of claim 1, wherein said main memory iscached in a CPU cache or an external cache.
 4. The method of claim 1,wherein copying of a selected subset of the entire image from the I/ORAM or copying of the processed image into the I/O RAM comprises copyingusing DMA circuitry.
 5. The method of claim 1, wherein said I/O RAM is adisplay video RAM associated with a video output device configured todisplay the copy of the processed image.
 6. The method of claim 1,wherein a pixel comprises a plurality of color components, a colorcomponent comprises a number of color component bits, and the pluralityof pixel bits of a pixel comprises the color component bits of the colorcomponents, and wherein extracting a subset of pixel bits from a pixelcomprises extracting respective most significant color component bits ofthe color components of the pixel.
 7. The method of claim 6, whereinperforming CPU intensive operations further comprises encoding thevarious pixel values, wherein a data field in a data element comprises apixel value in an encoded form.
 8. The method of claim 6, whereinperforming CPU intensive operations further comprises variably selectingone of 3, 4, 5, 8, 9, 12, 15, 16, and 24, as the number of pixel bits tosub-sample from a pixel, wherein the number of pixel bits sub-sampledfrom a pixel is less than a number of pixel bits of the pixel beingbit-wise sub-sampled.
 9. The method of claim 6, wherein performing CPUintensive operations further comprises receiving a specification of thenumber of pixel bits to sub-sample from a pixel.
 10. The method of claim9, wherein performing CPU intensive operations further comprises:setting a size of the data field to the specification of the number ofpixel bits to sub-sample; and subtracting the size of the data fieldfrom a supported data size to derive a size of the run-length field. 11.A machine for image processing comprising: an I/O device including anI/O RAM for storing an entire image; a buffer of a main memory forstoring-a copy of a selected subset of the entire image; means forcopying the selected subset of the entire image from the I/O RAM to themain memory to generate the copy of the selected subset by multiplecalls to a memory copy function to copy each image line of the selectedsubset to a buffer line of the buffer; means for performing computingintensive processing on-the copy of the selected subset to generate aprocessed image and to store the processed image in the buffer; andmeans for copying the processed image from said main memory to the I/ORAM to generate a copy of the processed image by multiple calls to thememory copy function to copy each buffer line of the buffer to an imageline in the I/O RAM; wherein means for performing CPU intensiveoperations comprises means for bit-wise sub-sampling pixels of the copyselected subset to extract corresponding subsets of pixel bits from thepixels; means for counting consecutive repeated instances of variouspixel values represented by the extracted subsets of pixel bits; andmeans for outputting a series of data elements for the pixels to formthe processed image, based on the extracted subsets of pixel bits;wherein a data element comprises a data field and a run-length field,wherein the data field comprises a pixel value in a range spanning fromzero to a maximum value represent-able by a number of pixel bitssub-sampled from a pixel, and wherein the run-length field comprises aconsecutive repeat count of the pixel value in the data field.
 12. Themachine of claim 11, wherein the entire image comprises a televisionvideo image including a visible portion visible on a display device anda non-visible portion not visible on the display device and the selectedsubset of the entire image substantially comprises only the visibleportion.
 13. The machine of claim 11, wherein said I/O RAM is a capturevideo RAM.
 14. The machine of claim 11, wherein said I/O device furthercomprises means for outputting the copy of the processed image.
 15. Themachine of claim 11, wherein the main memory comprises cache memory tostore the copy of selected subset and the processed image.
 16. Themachine of claim 11, wherein a pixel comprises a plurality of colorcomponents, a color component comprises a number of color componentbits, and the plurality of pixel bits of a pixel comprises the colorcomponent bits of the color components, and wherein extracting a subsetof pixel bits from a pixel comprises extracting respective mostsignificant color component bits of the color components of the pixel.17. The machine of claim 16, wherein means for performing CPU intensiveoperations further comprises means for encoding the various pixelvalues, wherein a data field in a data element comprises a pixel valuein an encoded form.
 18. The machine of claim 16, wherein means forperforming CPU intensive operations further comprises means for variablyselecting one of 3, 4, 5, 8, 9, 12, 15, 16, and 24, as the number ofpixel bits to sub-sample from a pixel, wherein the number of pixel bitssub-sampled from a pixel is less than a number of pixel bits of thepixel being bit-wise sub-sampled.
 19. The machine of claim 16, whereinmeans for performing CPU intensive operations further comprises meansfor receiving a specification of the number of pixel bits to sub-samplefrom a pixel.
 20. The machine of claim 19, wherein means for performingCPU intensive operations further comprises: means for setting a size ofthe data field to the specification of the number of pixel bits tosub-sample; and means for subtracting the size of the data field from asupported data size to derive a size of the run-length field.
 21. Amachine for image processing comprising: an image input deviceconfigured to capture an entire image and including a capture video RAMto store the entire image; a buffer in a main memory to store a copy ofa selected subset of the entire image, and image planes separated fromthe selected subset of the entire image; means for copying the selectedsubset from the capture video RAM to the buffer to generate a copy ofthe selected subset by multiple calls to a memory copy function to copyeach image line of the selected subset to a buffer line of the buffer;means for separating the copy of the selected subset in the buffer intoa plurality of image planes; means for compressing the image planesusing different compression methods to generate a plurality ofcompressed image planes for storage in the main memory; and means forcopying the compressed image planes from the main memory to the capturevideo RAM by multiple calls to the memory copy function copying eachbuffer line of the buffer to generate an image line of a copy of acompressed image plane in the capture video RAM.
 22. The machine ofclaim 21, wherein the entire image comprises a television video imageincluding a visible portion visible on a display device and anon-visible portion not visible on the display device, and the selectedsubset of the entire image substantially comprises only the visibleportion.
 23. The machine of claim 21, wherein the main memory comprisescache memory to store the entire image, the uncompressed image planesand the compressed image planes.
 24. The machine of claim 21, whereinsaid means for separating comprises means for separating a first set ofpixels of the copy of the selected subset having a distinguishingcharacteristic into a first image plane, and means for separating asecond set of pixels of the copy of the selected subset not having thedistinguishing characteristic into a second image plane.
 25. The machineof claim 24, wherein the distinguishing characteristic comprises aneffective color depth, a rate of pixel value change between adjacentpixels, a marker or an indicator indicating selection by a user,associated with the first set of pixels.
 26. The machine of claim 21,wherein said means for separating comprises means for separatinggrayscale pixel values of the copy of the selected subset into one ormore grayscale image planes, and means for separating Doppler enhancedpixel values of the copy of the selected subset into one or more Dopplerenhanced image planes.
 27. The machine of claim 21, wherein said meansfor compressing comprises first means for compressing a first imageplane using a first compression method, and second means for compressinga second image plane using a second compression method, wherein thefirst and second compression methods are different compression methods.28. An image processing method comprising: copying a selected subset ofan entire image from a capture video RAM to a buffer of a main memory togenerate a copy of the selected subset by multiple calls to a memorycopy function to copy each image line of the selected subset to a bufferline of the buffer; separating the copy of the selected subset in thebuffer into a plurality of image planes; compressing the image planesusing different compression methods to generate a plurality ofcompressed image planes for storage in the main memory; and copying thecompressed image planes from the main memory by multiple calls to thememory copy function copying each buffer line of the buffer to generatean image line of a copy of a compressed image plane in the capture videoRAM.
 29. The method of claim 28, wherein the entire image comprises atelevision video image including a visible portion visible on a displaydevice and a non-visible portion not visible on the display device, andthe selected subset of the entire image substantially comprises only thevisible portion.
 30. The method of claim 28, wherein separatingcomprises separating a first set of pixels of the copy of the selectedsubset having a distinguishing characteristic into a first image plane,and separating a second set of pixels of the copy of the selected subsetnot having the distinguishing characteristic into a second image plane.31. The method of claim 30, wherein the distinguishing characteristiccomprises an effective color depth, a rate of pixel value change betweenadjacent pixels, a marker or an indicator indicating selection by auser, associated with the first set of pixels.
 32. The method of claim28, wherein separating comprises separating grayscale pixel values ofthe copy of the selected subset into one or more grayscale image planes,and separating Doppler enhanced pixel values of the copy of the selectedsubset into one or more Doppler enhanced image planes.
 33. The method ofclaim 28, wherein compressing comprises compressing a first image planeusing a first compression method, and compressing a second image planeusing a second compression method, wherein the first and secondcompression methods are different compression methods.